Try this notebook in Binder.

Inspecting GRIB data

When running Metview through its graphical user interface, the easiest way to inspect a GRIB file is to right-click on it and choose ‘examine’ from the context menu. Similar functionality is available from the command line: “metview -e /path/to/grib/file”.

This notebook shows how to inspect GRIB data from within a Python environment. Note that filtering, processing and visualising data is covered in other notebooks and gallery examples.

[1]:
import metview as mv
[2]:
filename = "data_fc.grib"
if mv.exist(filename):
    fs = mv.read(filename)
else:
    fs = mv.gallery.load_dataset(filename)

Data overview

In Metview, GRIB data is represented by a Fieldset object. Many Metview operations generate GRIB data, we can retrieve it from MARS, or we can load it directly from disk.

GRIB files are composed of messages, one for each field. The describe() method groups the GRIB messages by parameter and gives a summary for quick inspection. Here we will see that the Fieldset contains parameters q,t,u,v,z for multiple vertical levels, and tp at surface level at various forecast steps. We will see in a moment what these parameters actually are in case you’re not familiar with the letters!

[3]:
fs.describe()
[3]:
parametertypeOfLevelleveldatetimestepparamIdclassstreamtypeexperimentVersionNumber
qisobaricInhPa100,250,...2011121500,6,...133odoperfc0001
tisobaricInhPa100,250,...2011121500,6,...130odoperfc0001
tpsurface02011121500,6,...228odoperfc0001
uisobaricInhPa100,250,...2011121500,6,...131odoperfc0001
visobaricInhPa100,250,...2011121500,6,...132odoperfc0001
zisobaricInhPa100,250,...2011121500,6,...129odoperfc0001

We can look a bit deeper into each parameter via its shortName, for example, temperature:

[4]:
fs.describe("t")
[4]:
shortNamet
paramId130
typeOfLevelisobaricInhPa
level100,250,300,500,700,850,1000
date20111215
time0
step0,6,12,18,24,30,36,42
classod
streamoper
typefc
experimentVersionNumber0001

Note that the data do not necessarily form a complete hypercube - this summary lists all the values for each ‘dimension’, but it is not necessarily the case that, for example, all the steps exist for all the levels.

We can also specify the parameter via its numeric paramId, for example, total precipitation:

[5]:
fs.describe(228)
[5]:
shortNametp
paramId228
typeOfLevelsurface
level0
date20111215
time0
step0,6,12,18,24,30,36,42
classod
streamoper
typefc
experimentVersionNumber0001

Data table

fs is a Fieldset object representing the GRIB file. It is essentially a list of GRIB messages. The ls() method displays information on a per-message basis. We can call it with the whole Fieldset, but that gives us a very long listing! So let’s call it on the first 10 fields using Python’s ‘slice’ operator:

[6]:
fs[:10].ls()
[6]:
centre shortName typeOfLevel level dataDate dataTime stepRange dataType gridType
Message
0 ecmf t isobaricInhPa 1000 20111215 0 0 fc regular_ll
1 ecmf z isobaricInhPa 1000 20111215 0 0 fc regular_ll
2 ecmf u isobaricInhPa 1000 20111215 0 0 fc regular_ll
3 ecmf v isobaricInhPa 1000 20111215 0 0 fc regular_ll
4 ecmf q isobaricInhPa 1000 20111215 0 0 fc regular_ll
5 ecmf t isobaricInhPa 850 20111215 0 0 fc regular_ll
6 ecmf z isobaricInhPa 850 20111215 0 0 fc regular_ll
7 ecmf u isobaricInhPa 850 20111215 0 0 fc regular_ll
8 ecmf v isobaricInhPa 850 20111215 0 0 fc regular_ll
9 ecmf q isobaricInhPa 850 20111215 0 0 fc regular_ll

We can select a different range of fields and add extra ecCodes keys to get more information:

[7]:
fs[210:220].ls(extra_keys=['name', 'units'])
[7]:
centre shortName typeOfLevel level dataDate dataTime stepRange dataType gridType name units
Message
0 ecmf t isobaricInhPa 1000 20111215 0 36 fc regular_ll Temperature K
1 ecmf z isobaricInhPa 1000 20111215 0 36 fc regular_ll Geopotential m**2 s**-2
2 ecmf u isobaricInhPa 1000 20111215 0 36 fc regular_ll U component of wind m s**-1
3 ecmf v isobaricInhPa 1000 20111215 0 36 fc regular_ll V component of wind m s**-1
4 ecmf q isobaricInhPa 1000 20111215 0 36 fc regular_ll Specific humidity kg kg**-1
5 ecmf t isobaricInhPa 850 20111215 0 36 fc regular_ll Temperature K
6 ecmf z isobaricInhPa 850 20111215 0 36 fc regular_ll Geopotential m**2 s**-2
7 ecmf u isobaricInhPa 850 20111215 0 36 fc regular_ll U component of wind m s**-1
8 ecmf v isobaricInhPa 850 20111215 0 36 fc regular_ll V component of wind m s**-1
9 ecmf q isobaricInhPa 850 20111215 0 36 fc regular_ll Specific humidity kg kg**-1

We can add some keys that compute statistics per field (this will take longer, as the values will be decoded and the statistics computed - they are not stored in the GRIB header).

[8]:
fs[210:220].ls(extra_keys=['min', 'max'])
[8]:
centre shortName typeOfLevel level dataDate dataTime stepRange dataType gridType min max
Message
0 ecmf t isobaricInhPa 1000 20111215 0 36 fc regular_ll 250.973 312.973
1 ecmf z isobaricInhPa 1000 20111215 0 36 fc regular_ll -2506.38 3061.62
2 ecmf u isobaricInhPa 1000 20111215 0 36 fc regular_ll -14.0416 19.7084
3 ecmf v isobaricInhPa 1000 20111215 0 36 fc regular_ll -16.6044 19.3956
4 ecmf q isobaricInhPa 1000 20111215 0 36 fc regular_ll 8.95455e-05 0.0198649
5 ecmf t isobaricInhPa 850 20111215 0 36 fc regular_ll 245.139 302.639
6 ecmf z isobaricInhPa 850 20111215 0 36 fc regular_ll 10344.5 16136.5
7 ecmf u isobaricInhPa 850 20111215 0 36 fc regular_ll -16.2513 26.9987
8 ecmf v isobaricInhPa 850 20111215 0 36 fc regular_ll -21.4716 25.5284
9 ecmf q isobaricInhPa 850 20111215 0 36 fc regular_ll 8.98526e-05 0.0155317

We can also use type qualifiers (s=string, l=long, d=double) to give more control over how the keys are displayed:

[10]:
fs[210:220].ls(extra_keys=['centre:l'])
[10]:
centre shortName typeOfLevel level dataDate dataTime stepRange dataType gridType centre:l
Message
0 ecmf t isobaricInhPa 1000 20111215 0 36 fc regular_ll 98.0
1 ecmf z isobaricInhPa 1000 20111215 0 36 fc regular_ll 98.0
2 ecmf u isobaricInhPa 1000 20111215 0 36 fc regular_ll 98.0
3 ecmf v isobaricInhPa 1000 20111215 0 36 fc regular_ll 98.0
4 ecmf q isobaricInhPa 1000 20111215 0 36 fc regular_ll 98.0
5 ecmf t isobaricInhPa 850 20111215 0 36 fc regular_ll 98.0
6 ecmf z isobaricInhPa 850 20111215 0 36 fc regular_ll 98.0
7 ecmf u isobaricInhPa 850 20111215 0 36 fc regular_ll 98.0
8 ecmf v isobaricInhPa 850 20111215 0 36 fc regular_ll 98.0
9 ecmf q isobaricInhPa 850 20111215 0 36 fc regular_ll 98.0

We can add a filter to show only those messages whose keys match all the given conditions. Note that these must be valid ecCodes keys - ‘shortName’ or ‘paramId’ can be used for the parameter, but ‘param’ cannot, since it is not a key. Keys do not need to be displayed in order to be used as a filter.

[27]:
fs.ls(filter={'shortName':'t', 'step':6, 'centre:s':'ecmf'})
[27]:
centre shortName typeOfLevel level dataDate dataTime stepRange dataType gridType
Message
35 ecmf t isobaricInhPa 1000 20111215 0 6 fc regular_ll
40 ecmf t isobaricInhPa 850 20111215 0 6 fc regular_ll
45 ecmf t isobaricInhPa 700 20111215 0 6 fc regular_ll
50 ecmf t isobaricInhPa 500 20111215 0 6 fc regular_ll
55 ecmf t isobaricInhPa 300 20111215 0 6 fc regular_ll
60 ecmf t isobaricInhPa 250 20111215 0 6 fc regular_ll
65 ecmf t isobaricInhPa 100 20111215 0 6 fc regular_ll

Note that the output from ls() is a pandas dataframe, allowing further manipulation:

[12]:
p = fs[:10].ls()
print(type(p))
print(p.columns)
print(p['shortName'])
<class 'pandas.core.frame.DataFrame'>
Index(['centre', 'shortName', 'typeOfLevel', 'level', 'dataDate', 'dataTime',
       'stepRange', 'dataType', 'gridType'],
      dtype='object')
Message
0    t
1    z
2    u
3    v
4    q
5    t
6    z
7    u
8    v
9    q
Name: shortName, dtype: object

Getting more specific keys

To extract just the keys you want from the GRIB fields, use grib_get_long(), grib_get_double() and grib_get_string() to extract single keys, and grib_get() for multiple keys. These examples show how.

[13]:
fs[:5].grib_get_string('centre')
[13]:
['ecmf', 'ecmf', 'ecmf', 'ecmf', 'ecmf']
[15]:
fs[:5].grib_get_long('centre')
[15]:
[98.0, 98.0, 98.0, 98.0, 98.0]
[20]:
fs[43:48].grib_get_double('level')
[20]:
[850.0, 850.0, 700.0, 700.0, 700.0]

When extracting mutliple keys, it is more efficient to use grib_get() to get them all in one go:

[23]:
fs[43:48].grib_get(['centre', 'step', 'level'])
[23]:
[['ecmf', '6', '850'],
 ['ecmf', '6', '850'],
 ['ecmf', '6', '700'],
 ['ecmf', '6', '700'],
 ['ecmf', '6', '700']]

Since the grib_get() method does not specify a type for each key, we can add :l, :d or :s to each one:

[25]:
fs[43:48].grib_get(['centre:l', 'step', 'level'])
[25]:
[[98.0, '6', '850'],
 [98.0, '6', '850'],
 [98.0, '6', '700'],
 [98.0, '6', '700'],
 [98.0, '6', '700']]

The above command will return a list per field. We can instead tell it to group the results by key:

[26]:
fs[43:48].grib_get(['centre:l', 'step', 'level'], 'key')
[26]:
[[98.0, 98.0, 98.0, 98.0, 98.0],
 ['6', '6', '6', '6', '6'],
 ['850', '850', '700', '700', '700']]

Quick plot

Let’s plot the first field to see the geographic extents of the data:

[34]:
mv.setoutput('jupyter')
mv.plot(fs[0], mv.mcont(contour_automatic_setting='ecmwf'))
[34]:
../_images/examples_inspecting_grib_data_38_0.png
[ ]: