Test-suite

Installation

The test suite is available on GitHub via git

%git clone https://github.com/yambo-code/yambo-tests.git yambo-tests

you may need to provide your GitHub username and password.

After the checkout the first thing to do is the configuration

%cd test-suite
%./configure --with-yambo=/home/marini/Yambo/yambo/master/

If the configuration works you should see something similar to the following message:

%./configure --with-yambo=/home/marini/Yambo/yambo/master/
%checking build system type... x86_64-unknown-linux-gnu
%checking host system type... x86_64-unknown-linux-gnu
%checking the anomaly.mlib.ism.cnr.it ROBOT... created
%checking for nccopy... no
%checking for ncftp... ncftp
%checking for ncftpls... ncftpls
%checking for ncftpput... ncftpput
%checking for awk... awk
%checking for txt2html... txt2html
%checking the Yambo source... /home/marini/Yambo/yambo/master/
%configure: creating ./config.status
%config.status: creating config/MODULES.pl
%config.status: creating config/TOOLS.pl

Robot informations =

By typing

%./driver.pl -i

you will see some basic info about your machine. This command is useful to see the available flows, configurations and so on.

Database download

Now it is time to download all databases. This can be easily done using the -d option

%./driver.pl -d all

Auto-Testing

Compilation

By default the configure copies a basic compilation script in ROBOTS/<YOUR ROBOT>/CONFIGURATIONS/default.sh. This uses internally compiled libraries so it should work easily. In order to test it launch

%./driver.pl -compile

Internal Test

Use the autotest to verify your compiled source:

%./driver.pl -autotest

If everything works you should see as log the following message

%==================================================================================
%= Starting Yambo test-suite
%==================================================================================
% - Check requirements : ...ncftp...ncftpls...ncftpput...txt2html
% -     Test selection : from input
%               ncdump : /opt/gcc4/netcdf-4.0.1/bin/ncdump
% -  Executable checks : (yambo  OK ) (ypp  FAIL )(a2y  FAIL )(p2y  FAIL )
%        Running tests : hBN/GW-OPTICS 01_init 02_HF
%             Projects : nopj
%            Verbosity : low 
%            Precision : 0.01 
%             Hostname : anomaly.mlib.ism.cnr.it 
%             revision : 14505 
%                build : MPI+SLK 
%              bin dir : bin 
%           shortname  : 
%               source : master_-precompiled 
%        Parallel Loop : SERIAL
%=--------------------------------------------------------------------------------=
% >  [1  /1  ] hBN/GW-OPTICS[serial]   5 passes 
%=--------------------------------------------------------------------------------=

The 5 passes are important as they represent the number of passed tests.

For the impatiens: quick cron run and final reporting

If you do not want to bother with all details of the testing script and want just to install the suite read this section and stop.

In order to properly setup your machine for night test-suite runs you have to:

1. Choose the branch(s) to test

For this purpose use

%./driver.pl -edit branches

1. create a crontab entry

See here for detailled informations on cron. My personal crontab file looks like

%# m  h   dom mon dow   command
% 30 18  *   *   Thu   ~/bin/scripts/yambo_validation.tcsh validate >> /home/marini/Yambo/sources/svn/yambo-tests/test-suite/LOG 2>&1

with

%>cat bin/scripts/yambo_validation.tcsh 
%#!/usr/bin/tcsh
%cd /home/marini/Yambo/sources/svn/yambo-tests/test-suite
%./driver.pl -flow $argv[1] -report

That's all you need.

Available tests

The list of tests can be done using the -l option:

%./driver.pl -l

Note that you can add a TEST to -l to list the specific inputs of that test. For example

%>./driver.pl -l GaAs001-c4x4/YAMBO
%
%Available input files for GaAs001-c4x4/YAMBO are:  01_init 02_RAS_full 03_RAS_full 04_RAS_front 05_RAS_dimer

Reports & Logs

The autotest and any other run of the test-suite produces three files. Each file is labelled with the date of the run.

In order to see the meaning and the in formations written in the files let's do a quick run of a Dummy test created in order to reproduce some of the possible error conditions.

Let's first clean the suite

%./driver.pl -c

and use the -force to run also tests tagged as BROKEN (like the Dummy one). First thing to do is to download the Dummy database adding the -force option.

%./driver.pl -d Dummy -force

Then we can run the Dummy test using

%./driver.pl -tests Dummy -force

The result will now be

%=--------------------------------------------------------------------------------=
% >  [1  /1  ] Dummy[serial]          11 passes  5 fails 
%=--------------------------------------------------------------------------------=

The 5 fails cover most of the potential errors detected by the test-suite. Details can be now found in the report/log files

REPORT file

This file contains very condensed information about the run. There is the duration time plus some detail of the reasons behind the 5 fails.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Yambo test suite global REPORT
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%==================================================================================
%Running tests           :Dummy
%Projects                :nopj
%Date - Time             :Fri-19-May - 11-46
%Build                   :master - MPI+SLK - rev.14505
%Parallel Conf           :[serial] - none
%=--------------------------------------------------------------------------------=
%FAIL: 5 & NO RUN: 0 & SKIPS: 0 % WHITELIST: 1 % OKs: 10
%FAIL details: 1 wrong output % 1 no reference % 1 no output % 2 no DB
%=--------------------------------------------------------------------------------=
%% Section Duration : 0:0:3 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% TOTAL   Duration : 0:0:3 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

In the dummy case indeed the suite should detect:

1. 1 wrong output
2. 1 output missing the REFERENCE
3. 1 missing output (not generated by the run)
4. 2 missing databases.

More details can be found in the LOG and and in the ERROR.

The REPORT file is the one used to send final logs to the mailing list.

ERROR file

This is a useful file with detailed information about the errors only.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Yambo test suite ERRORs log
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%==================================================================================
%Running tests           :Dummy
%Projects                :nopj
%Date - Time             :Fri-19-May - 11-46
%Build                   :master - MPI+SLK - rev.14505
%Parallel Conf           :[serial] - none
%=--------------------------------------------------------------------------------=
%Dummy/02_hf-serial-master/o-02_hf.hf  : VALUE(#  17)@COL#  6 is  -5.4691     [REF] vs  -4.4691     [OUT] corresponding to  6.1013     [% RELATIVE TO MAX]
%Dummy/02_hf-serial-master  :  NO REFERENCE/o-02_hf.ndb.em1s_fragment_1 in OUTPUT
%Dummy/04_em1s-serial-master  :  NO o-04_em1s.ndb.em1s_fragment_1 in REFERENCE
%% Section Duration : 0:0:3 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% TOTAL   Duration : 0:0:3 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Running

Before entering in the details of the different running methods let's first describe the Parallel Options

Parallel Options

%   (parallel options)
%             -np     <N>            Fixed number of CPU used
%             -np_min <N>            Minimum number of CPU used
%             -np_max <N>            Maximum number of CPU used
%             -nt     <T>            # of OMP threads
%             -nl     <L>            # of CPU used for linear algebra
%             -def_par               Use the default parallelization scheme
%             -rand_par              Use randomly generated parallel structures

Examples:

%./driver.pl -tests Dummy -force -np 2

produces all possible combinations of 2 cpu's for the Dummy tests. Each run is labelled in such a way to be unique. The Dummy folder will look like

%%l tests/Dummy/
%01_init-Nmpi2-1-master/  02_hf-Nmpi2-3-master/      03_optics-Nmpi2-3-master/  04_em1s-Nmpi2-3-master/  INPUTS/     yambo.in
%02_hf                    03_optics                  04_em1s                    04_em1s-Nmpi2-4-master/  LOG/
%02_hf-Nmpi2-1-master/    03_optics-Nmpi2-1-master/  04_em1s-Nmpi2-1-master/    BROKEN                   REFERENCE/
%02_hf-Nmpi2-2-master/    03_optics-Nmpi2-2-master/  04_em1s-Nmpi2-2-master/    Dummy.tar.gz             SAVE/

Note that each run is labelled by the number of MPI cpus (Nmpi2) and by the increasing counter of potential combinations of the two cpu's (-1-master,-2-master,...).

Other potential combinations are

• MPI, default parallelization (no cycle)

%./driver.pl -tests Dummy -force -np 2 -def_par
 

• MPI, random parallelization (no cycle, good for testing)

%./driver.pl -tests Dummy -force -np 2 -rand_par

• MPI, using 2 and 4 cpu's, no cycle on combinations, using random parallelization

%./driver.pl -tests Dummy -force -np_min 2 -np_max 4 -rand_par

• OpenMP (if compiled with OpenMP)

%./driver.pl -tests Dummy -force -nt 2

• MPI+SLK

%./driver.pl -tests Dummy -force -np 4 -nl 4 -def_par

Tests selection

The test-suite can be run in three different ways

1. Using a theme
2. Passing individual sets of tests
3. Using flows

Moreover tests can be selected using projects.

Flows are discussed in the Flows.

Theme

A theme is a just a series of tests. Themes are stored in the THEMES/ folder and are used simply using

%./driver.pl -theme test

test is a text file in the THEMES folder.

Individual Tests

This is the way we used to run the Dummy test. Just use

%./driver.pl -tests Dummy

to select a combination of specific inputs and tests you can use more complex tests options:

%./driver.pl -tests "Dummy 01_init 02_hf; Si_bulk/GW-OPTICS 00_init 02Cohsex"

Projects & Keys

Project are listed with the test lists and are accessed with the lowercase options: elph,sc,rt,nl...

So in order to run all tests belonging to the sc project just type

%./driver.pl -tests all -keys sc

Actually the -keys option is more powerful as, in the future, should also select all inputs with specific observables (like gw, optics and so on).

The list of project are accessed by simply the first column in the list obtained by using the -l option. In the following are in bold.

%%./driver.pl -l
%Available test materials/systems: 
%[YAMBO]    Al111
%           C6H3Cl3
%           PA_chain[H]
%           SiH4
%           Al_bulk/GW-OPTICS
%           H2/OPTICS
%           H_chain/2.05
%           H_chain/2.5
%           hBN/GW-OPTICS
%           LiF/GW-OPTICS
%           Si_bulk/GW-OPTICS
%           Si_surface/OPTICS
%           Si_wire/OPTICS
%           MoS2/pwscf/OPTICS
%           MoS2/abinit/OPTICS/With-SOC
%           MoS2/abinit/OPTICS/Without-SOC-sp1
%           MoS2/abinit/OPTICS/Without-SOC-sp2
%[QED]      Si_bulk/QED
%[NL]       hBN/NL
%[RT]       AlAs[H]
%           H2/RT
%           hBN/RT
%           Si_bulk/RT
%           WSe2/RT[H]
%           MoS2/pwscf/RT
%[SC]       H2/SC
%           hBN/SC
%           Si_bulk/SC
%[MAGNETIC] Si_bulk/MAGNETIC
%[ELPH]     Diamond/ELPH1[H]
%           Si_bulk/ELPH/BSE
%           Si_bulk/ELPH/ELPH_base
%           Si_bulk/ELPH/ELPH_for_BSE
%           Si_bulk/ELPH/QP_CTL
%[KERR]     Cobalt
%           Nickel
%           Iron/pwscf
%           Iron/abinit/With-SOC
%           Iron/abinit/Without-SOC
%[BROKEN]   Dummy
%           GaAs001-c4x4/YAMBO
%           hBN/PL[PL]
%           Si001-c4x2/YAMBO
%           WS2/PL[PL]

Note that to access any of these projects you must use the lowercase option. For example

• SC → -keys sc
• RT → -keys rt
• SC+RT → -keys 'sc rt'

HARD tests

Some of the tests are long. In the tests list they are labelled with a [H]. In order to run them the key hard must be used. For example

%./driver.pl -tests all -keys 'sc rt hard'

runs all SC+RT tests including the heavy ones.

FLOWS

All properties of a single launch of the test-suite and also a more complex test-suite launch looping on several parallel, tests, branches, combinations can be controlled via the FLOW feature.

A FLOW is a perl file (like validate.pl) containing combinations of descriptors.

All FLOW files are in the ROBOTS/<YOUR ROBOT>/FLOWS.

A FLOW is run using

%./driver.pl -flow validate

A general flow descriptor is

% {
% ACTIVE      => "yes", # can be yes or no
% MPI_CPU     => "NP",
% SLK_CPU     => "NM",
% PAR_MODE    => "TEXT", # (TEXT can be default, random, loop)
% THREADS     => "NT",
% BRANCH      => "/home/marini/Yambo/sources/git/yambo/branches/SLK", # (SAVED)
% THEME       => "G_parallelization", # (SAVED)
% CONFIG      => "gfortran_slk.sh", # (SAVED)
% TESTS       => "hBN/GW-OPTICS", #  (SAVED)
% KEYS        => "nopj elph hard bse rpa", # (SAVED)
%}

The fields labelled with a (SAVE) are not changed if this descriptor is included in a more complex FLOW. In order to explain this feature let's just have a look at the default validate FLOW provided by default:

%#!/usr/bin/perl
%#
%@flow = (
%{
% ACTIVE      => "yes",
% CONFIG      => "default.sh",
% KEYS        => "nopj elph sc hard",
%},
%{
% MPI_CPU     => $SYSTEM_NP,
% PAR_MODE    => "default",
%},
%{
% MPI_CPU     => $SYSTEM_NP,
% PAR_MODE    => "random",
%},
%{
% MPI_CPU     => $SYSTEM_NP_half,
% PAR_MODE    => "loop",
%},
%{
% THREADS     => $SYSTEM_NP_half,
%},
%{
% MPI_CPU     => $SYSTEM_NP,
% SLK_CPU     => $SYSTEM_NP_half,
% PAR_MODE    => "default",
%},
%{
% MPI_CPU     => $SYSTEM_NP_half,
% THREADS     => $SYSTEM_NP_half,
% SLK_CPU     => $SYSTEM_SLK,
% PAR_MODE    => "default",
%},
%);

When this flow is launched the driver.pl will run all tests of the "nopj elph sc hard" projects using: a serial run, all system CPU's with default parallelization, with random parallelization, cycling among all possible combinations and so so...

Note that also the -autotest is just a simple calculation flow.

Robots

In version 5.0 the concept of Robots was introduced. In practice each time you run the driver.pl test-suite in any form (compilation, flow, manual tests and so on) the perl script will create a ROBOT.

In practice this means that all operations will be labelled with a <ROBOT> string. Single tests, executables will be copied to a dedicated ROBOT folder and also the LOG and REPORT files will be labelled by the specific ROBOT they belong to.

The benefits of this ROBOT structure is that multiple ROBOTS can be run simultaneously on the same test-suite!

Try yourself by simply running two instances of driver.pl.

WARNING: when you run more than 1 robot at the same time be sure to give time to the driver.pl to open all log files.

In addition if you are compiling the same source leave the robot the time to compile it before running a second robot on the same source.

Robot specific operations

You can clean/kill only the robot ID by using

%./driver.pl -c -robot ID
%./driver.pl -kill -robot ID

You can also launch the driver.pl with a specific robot ID

%./driver.pl -flow validate -robot ID

Robot branches

When several robots are run at the same time it is often necessary to specify for each robot a branch to do. This can be easily done via the branch file. This is an example of robot dependent branch file (obtained via the ./driver.pl -e branches) command

/home/marini/yambo/master master R1
/home/marini/yambo/master master R2
/home/marini/yambo/gpl master R3
/home/marini/yambo/gpl master R4
/home/marini/yambo/gpl master R5
/home/marini/yambo/gpl master R6
/home/marini/yambo/branches/devel-rt-stable devel-rt-stable R7
/home/marini/yambo/branches/devel-rt-stable devel-rt-stable R8
                                                                 

This file commands the driver.pl to run the master when the robot ID (-robot ID) is 1 or 2, the gpl for robots from 3 to 6 and so on.