Test-suite

Web-Interface

The updated list of test-suite runs can be found on the old Yambo web-page.

Tests' list is splitted in branches. For each branch many robots are shown. Check the bug-fixes branch as an example.

This list is automatically updated by the test-suite driver.pl when the -report option is used.

Installation

The test suite is available on GitHub via git

$git clone https://github.com/yambo-code/yambo-tests.git yambo-tests

you may need to provide your GitHub username and password.

Then init and update the ROBOTS submodule

$cd yambo-tests
$git submodule init
$git submodule update
$cd ROBOTS
$git checkout master
$cd ..

After the first thing to do is the configuration

$./configure --with-yambo=/home/marini/Yambo/yambo/master/

If the configuration works you should see something similar to the following message:

$./configure --with-yambo=/home/marini/Yambo/yambo/master/
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking the anomaly.mlib.ism.cnr.it ROBOT... created
hecking for nccopy... no
checking for ncftp... ncftp
checking for ncftpls... ncftpls
checking for ncftpput... ncftpput
checking for awk... awk
checking for txt2html... txt2html
checking the Yambo source... /home/marini/Yambo/yambo/master/
configure: creating ./config.status
config.status: creating config/MODULES.pl
config.status: creating config/TOOLS.pl

Robot informations

By typing

$./driver.pl -i

you will see some basic info about your machine. This command is useful to see the available flows, configurations and so on. An example is

  Running on host: anomaly.mlib.ism.cnr.it
  By user        : marini
  Version        : 5.0
  Available CPU's: 8
  Available confs: default.sh
  Available flows: failed_master.pl validate.pl default.pl failed_SLK.pl failed_devel-qp-dbs.pl minimal.pl

Database download

Now it is time to download all databases. This can be easily done using the -d option

$./driver.pl -d all

Compilation

By default the configure copies a basic compilation script in ROBOTS/<YOUR ROBOT>/CONFIGURATIONS/default.sh. This uses internally compiled libraries so it should work easily. In order to test it launch

$./driver.pl -compile

Internal Test

Use the autotest to verify your compiled source:

$./driver.pl -autotest

If everything works you should see as log the following message

==================================================================================
= Starting Yambo test-suite
==================================================================================
 - Check requirements : ...ncftp...ncftpls...ncftpput...txt2html
 -     Test selection : from input
               ncdump : /opt/gcc4/netcdf-4.0.1/bin/ncdump
 -  Executable checks : (yambo  OK ) (ypp  FAIL )(a2y  FAIL )(p2y  FAIL )
        Running tests : hBN/GW-OPTICS 01_init 02_HF
             Projects : nopj
            Verbosity : low 
            Precision : 0.01 
             Hostname : anomaly.mlib.ism.cnr.it 
             revision : 14505 
                build : MPI+SLK 
              bin dir : bin 
           shortname  : 
               source : master_-precompiled 
        Parallel Loop : SERIAL
=--------------------------------------------------------------------------------=
 >  [1  /1  ] hBN/GW-OPTICS[serial]   5 passes 
=--------------------------------------------------------------------------------=

The 5 passes are important as they represent the number of passed tests.

For the impatiens: quick cron run and final reporting

If you do not want to bother with all details of the testing script and want just to install the suite read this section and stop.

In order to properly setup your machine for night test-suite runs you have to:

1. Choose the branch(s) to test

For this purpose use

$./driver.pl -edit branches

1. create a crontab entry

See here for detailled informations on cron. My personal crontab file looks like

# m  h   dom mon dow   command
 30 18  *   *   Thu   ~/bin/scripts/yambo_validation.tcsh validate >> /home/marini/Yambo/sources/svn/yambo-tests/test-suite/LOG 2>&1

with

$cat bin/scripts/yambo_validation.tcsh 
#!/usr/bin/tcsh
cd /home/marini/Yambo/sources/svn/yambo-tests/test-suite
./driver.pl -flow $argv[1] -report

That's all you need.

Available tests

The list of tests can be done using the -l option:

$./driver.pl -l

Note that you can add a TEST to -l to list the specific inputs of that test. For example

$./driver.pl -l GaAs001-c4x4/YAMBO

Available input files for GaAs001-c4x4/YAMBO are:  01_init 02_RAS_full 03_RAS_full 04_RAS_front 05_RAS_dimer

Reports & Logs

The autotest and any other run of the test-suite produces three files. Each file is labelled with the date of the run.

In order to see the meaning and the in formations written in the files let's do a quick run of a Dummy test created in order to reproduce some of the possible error conditions.

Let's first clean the suite

$./driver.pl -c

and use the -force to run also tests tagged as BROKEN (like the Dummy one). First thing to do is to download the Dummy database adding the -force option.

$./driver.pl -d Dummy -force

Then we can run the Dummy test using

$./driver.pl -tests Dummy -force

The result will now be

=--------------------------------------------------------------------------------=
 >  [1  /1  ] Dummy[serial]          11 passes  5 fails 
=--------------------------------------------------------------------------------=

The 5 fails cover most of the potential errors detected by the test-suite. Details can be now found in the report/log files

REPORT file

This file contains very condensed information about the run. There is the duration time plus some detail of the reasons behind the 5 fails.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Yambo test suite global REPORT
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
==================================================================================
Running tests           :Dummy
Projects                :nopj
Date - Time             :Fri-19-May - 11-46
Build                   :master - MPI+SLK - rev.14505
Parallel Conf           :[serial] - none
=--------------------------------------------------------------------------------=
FAIL: 5 & NO RUN: 0 & SKIPS: 0 % WHITELIST: 1 % OKs: 10
FAIL details: 1 wrong output % 1 no reference % 1 no output % 2 no DB
=--------------------------------------------------------------------------------=
% Section Duration : 0:0:3 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% TOTAL   Duration : 0:0:3 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

In the dummy case indeed the suite should detect:

1. 1 wrong output
2. 1 output missing the REFERENCE
3. 1 missing output (not generated by the run)
4. 2 missing databases.

More details can be found in the LOG and and in the ERROR.

The REPORT file is the one used to send final logs to the mailing list.

ERROR file

This is a useful file with detailed information about the errors only.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Yambo test suite ERRORs log
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
==================================================================================
Running tests           :Dummy
Projects                :nopj
Date - Time             :Fri-19-May - 11-46
Build                   :master - MPI+SLK - rev.14505
Parallel Conf           :[serial] - none
=--------------------------------------------------------------------------------=
Dummy/02_hf-serial-master/o-02_hf.hf  : VALUE(#  17)@COL#  6 is  -5.4691     [REF] vs  -4.4691     [OUT] corresponding to  6.1013     [% RELATIVE TO MAX]
Dummy/02_hf-serial-master  :  NO REFERENCE/o-02_hf.ndb.em1s_fragment_1 in OUTPUT
Dummy/04_em1s-serial-master  :  NO o-04_em1s.ndb.em1s_fragment_1 in REFERENCE
% Section Duration : 0:0:3 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% TOTAL   Duration : 0:0:3 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Running

Before entering in the details of the different running methods let's first describe the Parallel Options

Parallel Options

   (parallel options)
             -np     <N>            Fixed number of CPU used
             -np_min <N>            Minimum number of CPU used
             -np_max <N>            Maximum number of CPU used
             -nt     <T>            # of OMP threads
             -nl     <L>            # of CPU used for linear algebra
             -def_par               Use the default parallelization scheme
             -rand_par              Use randomly generated parallel structures

Examples:

$./driver.pl -tests Dummy -force -np 2

produces all possible combinations of 2 cpu's for the Dummy tests. Each run is labelled in such a way to be unique. The Dummy folder will look like

$ls tests/Dummy/
01_init-Nmpi2-1-master/  02_hf-Nmpi2-3-master/      03_optics-Nmpi2-3-master/  04_em1s-Nmpi2-3-master/  INPUTS/     yambo.in
02_hf                    03_optics                  04_em1s                    04_em1s-Nmpi2-4-master/  LOG/
02_hf-Nmpi2-1-master/    03_optics-Nmpi2-1-master/  04_em1s-Nmpi2-1-master/    BROKEN                   REFERENCE/
02_hf-Nmpi2-2-master/    03_optics-Nmpi2-2-master/  04_em1s-Nmpi2-2-master/    Dummy.tar.gz             SAVE/

Note that each run is labelled by the number of MPI cpus (Nmpi2) and by the increasing counter of potential combinations of the two cpu's (-1-master,-2-master,...).

Other potential combinations are

• MPI, default parallelization (no cycle)

$./driver.pl -tests Dummy -force -np 2 -def_par
 

• MPI, random parallelization (no cycle, good for testing)

$./driver.pl -tests Dummy -force -np 2 -rand_par

• MPI, using 2 and 4 cpu's, no cycle on combinations, using random parallelization

$./driver.pl -tests Dummy -force -np_min 2 -np_max 4 -rand_par

• OpenMP (if compiled with OpenMP)

$./driver.pl -tests Dummy -force -nt 2

• MPI+SLK

$./driver.pl -tests Dummy -force -np 4 -nl 4 -def_par

Tests selection

The test-suite can be run in three different ways

1. Using a theme
2. Passing individual sets of tests
3. Using flows

Moreover tests can be selected using projects.

Flows are discussed in the Flows.

Theme

A theme is a just a series of tests. Themes are stored in the THEMES/ folder and are used simply using

%./driver.pl -theme test

test is a text file in the THEMES folder.

Individual Tests

This is the way we used to run the Dummy test. Just use

%./driver.pl -tests Dummy

to select a combination of specific inputs and tests you can use more complex tests options:

%./driver.pl -tests "Dummy 01_init 02_hf; Si_bulk/GW-OPTICS 00_init 02Cohsex"

Projects & Keys

Project are listed with the test lists and are accessed with the lowercase options: elph,sc,rt,nl...

So in order to run all tests belonging to the sc project just type

$./driver.pl -tests all -keys sc

Actually the -keys option is more powerful as, in the future, should also select all inputs with specific observables (like gw, optics and so on).

The list of project are accessed by simply the first column in the list obtained by using the -l option. In the following are in bold.

$./driver.pl -l
Available test materials/systems: 
[YAMBO]    Al111
           C6H3Cl3
           PA_chain[H]
           SiH4
           Al_bulk/GW-OPTICS
           H2/OPTICS
           H_chain/2.05
           H_chain/2.5
           hBN/GW-OPTICS
           LiF/GW-OPTICS
           Si_bulk/GW-OPTICS
           Si_surface/OPTICS
           Si_wire/OPTICS
           MoS2/pwscf/OPTICS
           MoS2/abinit/OPTICS/With-SOC
           MoS2/abinit/OPTICS/Without-SOC-sp1
           MoS2/abinit/OPTICS/Without-SOC-sp2
[QED]      Si_bulk/QED
[NL]       hBN/NL
[RT]       AlAs[H]
           H2/RT
           hBN/RT
           Si_bulk/RT
           WSe2/RT[H]
           MoS2/pwscf/RT
[SC]       H2/SC
           hBN/SC
           Si_bulk/SC
[MAGNETIC] Si_bulk/MAGNETIC
[ELPH]     Diamond/ELPH1[H]
           Si_bulk/ELPH/BSE
           Si_bulk/ELPH/ELPH_base
           Si_bulk/ELPH/ELPH_for_BSE
           Si_bulk/ELPH/QP_CTL
[KERR]     Cobalt
           Nickel
           Iron/pwscf
           Iron/abinit/With-SOC
           Iron/abinit/Without-SOC
[BROKEN]   Dummy
           GaAs001-c4x4/YAMBO
           hBN/PL[PL]
           Si001-c4x2/YAMBO
           WS2/PL[PL]

Note that to access any of these projects you must use the lowercase option. For example

• SC → -keys sc
• RT → -keys rt
• SC+RT → -keys 'sc rt'

HARD tests

Some of the tests are long. In the tests list they are labelled with a [H]. In order to run them the key hard must be used. For example

%./driver.pl -tests all -keys 'sc rt hard'

runs all SC+RT tests including the heavy ones.

FLOWS

All properties of a single launch of the test-suite and also a more complex test-suite launch looping on several parallel, tests, branches, combinations can be controlled via the FLOW feature.

A FLOW is a perl file (like validate.pl) containing combinations of descriptors.

All FLOW files are in the ROBOTS/<YOUR ROBOT>/FLOWS.

A FLOW is run using

%./driver.pl -flow validate

A general flow descriptor is

% {
% ACTIVE      => "yes", # can be yes or no
% MPI_CPU     => "NP",
% SLK_CPU     => "NM",
% PAR_MODE    => "TEXT", # (TEXT can be default, random, loop)
% THREADS     => "NT",
% BRANCH      => "/home/marini/Yambo/sources/git/yambo/branches/SLK", # (SAVED)
% THEME       => "G_parallelization", # (SAVED)
% CONFIG      => "gfortran_slk.sh", # (SAVED)
% TESTS       => "hBN/GW-OPTICS", #  (SAVED)
% KEYS        => "nopj elph hard bse rpa", # (SAVED)
%}

The fields labelled with a (SAVE) are not changed if this descriptor is included in a more complex FLOW. In order to explain this feature let's just have a look at the default validate FLOW provided by default:

%#!/usr/bin/perl
%#
%@flow = (
%{
% ACTIVE      => "yes",
% CONFIG      => "default.sh",
% KEYS        => "nopj elph sc hard",
%},
%{
% MPI_CPU     => $SYSTEM_NP,
% PAR_MODE    => "default",
%},
%{
% MPI_CPU     => $SYSTEM_NP,
% PAR_MODE    => "random",
%},
%{
% MPI_CPU     => $SYSTEM_NP_half,
% PAR_MODE    => "loop",
%},
%{
% THREADS     => $SYSTEM_NP_half,
%},
%{
% MPI_CPU     => $SYSTEM_NP,
% SLK_CPU     => $SYSTEM_NP_half,
% PAR_MODE    => "default",
%},
%{
% MPI_CPU     => $SYSTEM_NP_half,
% THREADS     => $SYSTEM_NP_half,
% SLK_CPU     => $SYSTEM_SLK,
% PAR_MODE    => "default",
%},
%);

When this flow is launched the driver.pl will run all tests of the "nopj elph sc hard" projects using: a serial run, all system CPU's with default parallelization, with random parallelization, cycling among all possible combinations and so so...

Note that also the -autotest is just a simple calculation flow.

Robots

In version 5.0 the concept of Robots was introduced. In practice each time you run the driver.pl test-suite in any form (compilation, flow, manual tests and so on) the perl script will create a ROBOT.

In practice this means that all operations will be labelled with a <ROBOT> string. Single tests, executables will be copied to a dedicated ROBOT folder and also the LOG and REPORT files will be labelled by the specific ROBOT they belong to.

The benefits of this ROBOT structure is that multiple ROBOTS can be run simultaneously on the same test-suite!

Try yourself by simply running two instances of driver.pl.

WARNING: when you run more than 1 robot at the same time be sure to give time to the driver.pl to open all log files.

In addition if you are compiling the same source leave the robot the time to compile it before running a second robot on the same source.

Robot specific operations

You can clean/kill only the robot ID by using

%./driver.pl -c -robot ID
%./driver.pl -kill -robot ID

You can also launch the driver.pl with a specific robot ID

%./driver.pl -flow validate -robot ID

Robot branches

When several robots are run at the same time it is often necessary to specify for each robot a branch to do. This can be easily done via the branch file. This is an example of robot dependent branch file (obtained via the ./driver.pl -e branches) command

/home/marini/yambo/master master R1
/home/marini/yambo/master master R2
/home/marini/yambo/gpl master R3
/home/marini/yambo/gpl master R4
/home/marini/yambo/gpl master R5
/home/marini/yambo/gpl master R6
/home/marini/yambo/branches/devel-rt-stable devel-rt-stable R7
/home/marini/yambo/branches/devel-rt-stable devel-rt-stable R8
                                                                 

This file commands the driver.pl to run the master when the robot ID (-robot ID) is 1 or 2, the gpl for robots from 3 to 6 and so on.

Adding New Tests

Adding a new test is easy.

Let's use a simple example.

Imagine I want to add the test hBN/PIPPO for which I have the SAVE and a list of input files. I copy everything in TESTS/MAIN/hBN creating a new directory, PIPPO

hBN>ls
GW-OPTICS  hBN.tar.gz  NL  PIPPO  PL	RT  SC

In PIPPO I have only the INPUTS and SAVE folders

PIPPO>ls -R
.:
INPUTS  SAVE  

./INPUTS:

01_init  02_HF

./SAVE:
ns.db1	ns.kb_pp  ns.kb_pp_fragment_1  ns.kb_pp_fragment_2  ns.kb_pp_fragment_3  ns.kb_pp_fragment_4  ns.wf  ns.wf_fragments_1_1  
ns.wf_fragments_2_1  ns.wf_fragments_3_1  ns.wf_fragments_4_1

At this point I simply run

./driver.pl -update hBN/PIPPO

The driver will run the inputs, create the REFERENCE folder, the .tar.gz of the new test which is uploaded on the ftp server.

Remember to add -compile if your source is not compiled.

At the end of the calculation the git status command should give

yambo-tests>git status
On branch master
Your branch is up-to-date with 'origin/master'.
Changes to be committed:
 (use "git reset HEAD <file>..." to unstage)
    new file:   TESTS/MAIN/hBN/PIPPO/INPUTS/01_init
    new file:   TESTS/MAIN/hBN/PIPPO/INPUTS/02_HF
    new file:   TESTS/MAIN/hBN/PIPPO/REFERENCE/l-02_HF_HF_and_locXC
    new file:   TESTS/MAIN/hBN/PIPPO/REFERENCE/o-02_HF.hf
    new file:   TESTS/MAIN/hBN/PIPPO/REFERENCE/o-02_HF.ndb.HF_and_locXC
    new file:   TESTS/MAIN/hBN/PIPPO/REFERENCE/r-02_HF_HF_and_locXC
    new file:   TESTS/MAIN/hBN/PIPPO/SAVE/.empty

It's now enough that you push and the new test will be ready!