1 Start the Evaluation System via Freva

... in the shell

To log in to the miklip system you may use ssh from any linux/unix system:

The -X will allow you to connect to the remote X server (e.g. to display images). user-account should be your account you usally login with, if you still don’t have one please ask the admins.

Start setting up the environment by loading the proper module (You may copy the following line as is into your shell):

This activates the system for your current session. You might notice some other modules have been loaded.

... in the web

To log in to the research group system you may use a browser:

The domain could be different to the shell domain, depending on how the admins set up the system. Use your normal user account.

2 Work with the Evaluation System via Freva

Freva is the all in one framework with the main features: freva –help

The core applications are the plugin, history and databrowser — same in web and shell.

Example Table

All commands have a -h or -help flag to display the commands help. Basic commands for Freva:

2.1 Plugins –plugins

This is the main access to all installed analysis tools and the history. The tools are implemented by providing plug-ins to the system. For more information on how to create a plugin check the Basic Developer Guide (BDG).

Basic Usage

To get the help:

To list all available analysis tools:

The ”Overview”:research.group.web/plugins of tools in the framework.

To select a particular tool:

You see here that the PCA tool is complaining because of an incomplete configuration.

To get the help of a particular tool:

Here you see the configuration parameter, its default value (None means there is no value setup), whether the configuration is mandatory ([mandatory] marking by the default value) and an explanation about the configuration parameter.

To pass the values to the tool you just need to use the key=value construct like this:

You may even define variables in terms of other variables like the projection name above. While doing so from the shell please remember you need to escape the $ sign by using the backslash (
) or setting the value in single quotes (no, double quotes don’t work). For example:

If you want to know more about this bash feature see this and if you want to want to know much more then take a look at this

Quoting is very important on any shell, so if you use them, be sure to know how it works. It may help you avoid losing data!

Configuring the plugins

All configurations are saved in the –history can be seen, saved, return into a command and restarted!

You may want to save the configuration of the tool:

Note this starts the tool. To just save the configuration without starting the tool use the -n or –dry-run flag. Also note this stores the configuration in a special directory structure so the system can find it again.

You can save the configuration somewhere else:

The configuration stored will be used to overwrite the default one. This is a possible usecase:

Change the defaults to suit your general needs:

Prepare some configurations you’ll be using recurrently

Scheduling

Instead of running your job directly in the terminal, you can involve the SLURM scheduler.

To run the tool murcss analyzing the variable tas the command is

The execution takes a certain time (here: roughly 1 minute) and prints

To schedule the same task you would use

instead. The output changes to

The last line shows you the command to view the output, which is created by the tool. In this example you would type

For jobs with a long run-time or large amounts of jobs you schould consider to schedule them and use the batch mode!

–help

2.2 History –history

To get the history in the web just click on ’History’ and browse around. In the shell:

It shows just the 10 latest entries, i.e. the 10 latest analysis that were performed. To create more complex queries check the help:

You can view the configuration used at nay time and the satatus of the created files (i.e. if the files are still there or has been modified)

The history offers a more direct way to re-run tools. The option return_command shows the analyze command belonging to the configuration. Here an example for the tool movieplotter:

It returns:

This is not an handy expression, but very useful. A re-run of the tool in batch shell could be easily performed by

2.3 Data-Browser –databrowser

All files available on the MiKlip server are scanned and indexed in a special server (SOLR). This allows us to query the server which responds almost immediately. Because of the miklip configuration the first time you call the tool it might take up to a couple of seconds to start. After that normally you should see results within a second.

2.3.1 Help

2.3.2 Usage

The databrowser expects a list of attribute=value (or key=value) pairs. There are a few differences and many more options (explained next).
Most important is that you don’t need to split the search according to the type of data you are searching for. You might as well search for files both on observations, reanalysis and model data all at the same time.

Also important is that all searches are made case insensitive (so don’t worry about upper or lower casing)

You can also search for attributes themselves instead of file paths. For example you can search for the list of variables available that satisfies a certain constraint (e.g. sampled 6hr, from a certain model, etc).

Defining the search

freva –databrowser project=baseline1 variable=tas time_frequency=mon

Defining the possible values

There are many more options for defining a value for a given attribute:

NOTE: When using * remember that your shell might give it a different meaning (normally it will try to match files with that name) to turn that off you can use backslash / in most shells

Regular Expressions must be given within forward slashes (/) and are match agains the whole value and not some part of it. Here’s a summary (there might be more... check it!) Syntax Meaning

Searching for metadata

You might as well want to now about possible values that an attribute can take after a certain search is done. For this you use the –facet flag (facets are the possible attributes that partition the result set). For example to see the time frequency (time resolution) in which reanalysis are available you might issue the following query:

You might also ask for more than one single facet by definig the –facet flag multiple times. For example let’s also see a list of variables:

Please note that those are not related, i.e. the values of the time_frequency facet do not correspond to any particular variable. It is like issuing to difference queries.

Also note that you can further define this as usual with a given query. For example check which files are at 6hr frequency:

If you want to see how many files would return if you further select that variable (drill down query) you may add the –count-facet-values flag (simply –count will also do):

This means that there are 7991 files containing the variable psl, 33 for sfcwind, and so on.

If you want to check all facets at once you may use the –all-facets flag (don’t worry this is still very fast)

And again you can also have the –count flag:

You might have also seen that some facets are not relevant at all as they are not partitioning the resulting data (e.g. see cmor_table or data_type). You can leave them out by adding the –relevant-only flag

If you try to retrieve all variables stored (remember there are over +2.100.000 files!) you’ll notice an ellipses (...) ath the end of the list:

This means there are more results than those being shown here. We limit the results to 100 for usability sake. If you still think this is a bug instead of a terrific feature, then you might use a special search word to change this facet.limit. That’s the number of results that will be retrieved. Setting it to -1 retrieves just everything... be aware that make cause some problems if you don’t know what you are doing (well sometimes it might also cause problems if you do... so use with discretion)

By the way, do you want to count them? Those are 619 variables!

Bash auto completion

And if that’s not awesome enough (I know it never is), then try the bash auto-completion. If you are using bash, everything is already setup when you issued the ’module load freva’ command. Whenever you hit tab the word will be completed to the longest unique string that matches your previous input. A second tab will bring up a list of all possible completions after that.

For example (?TAB? denotes presing the tab key):

results in

Now pressing ?TAB? again will show all other possibilities:

But flags are not the only thing being populated, it also work on atributes:

... and of course values:

And (yes! That wasn’t all) this is also query aware:

Note that if you mix flags this might not work as intended (or not at all).

2.4 –crawl_my_data

Per default it is loading the users ”projectdata” directory:
/research/database/data4project/projectdata/user-account

Help

Usage

would crawl all data you have in /research/database/data4project/projectdata/user-account

When you have a lot of data in your directory, it could be worth it, to take just a sub-directory. This gets much faster, when less data.

EXAMPLE: You’ve put in a new decadal experiment and just want to add this

2.5 –esgf

The search syntax is defined here: http://www.esgf.org/wiki/ESGF_Search_REST_API

It has been simplified to be used from the command line and resemble freva –databrowser as clos as possible. But the two commands rely on different backends which have different query possibilities.

Help

Usage

If you need some files: first check if they are there and how many they are:

You can check those urls by just not piping the result to wc (word count)

And you can get the wget script, a bash script written around wget to simplify data download using this:

By the way, the search looked for all files stored locally at DKRZ (distrib=false) holding the latest version (latest=true) of the variable tas (variable=tas) for the experiments decadal1960 to decadal1965 (this is a bash construct and not part of the search api!)