The Yambo Project - User contributions [en]

Test images

2022-06-01T09:04:12Z

Conor:

testing image upload
[[File:Stack4-rev2.png|thumb]]

[[File:K doping spin v2.png|thumb]]

File:K doping spin v2.png

2022-06-01T09:03:57Z

Conor:

Test images

2022-05-31T15:16:31Z

Conor: Created page with "testing image upload thumb"

testing image upload
[[File:Stack4-rev2.png|thumb]]

File:Stack4-rev2.png

2022-05-31T15:16:22Z

Conor:

just a test

Developers Corner

2022-05-31T15:14:42Z

Conor:

== Development Repositories ==
** [[Test-suite-simple|Test-suite (for dummies)]]
** [[Test-suite | Test-suite]]
** [[Benchmark-suite | Benchmark-suite]]
** [[Educational]]
* Special pages
** [[Special:AllPages | All Pages]]
** [[MediaWiki:Sidebar | Side Bar]]
** [[under construction | Html2Wiki HOWTO]]
** [https://en.wikipedia.org/wiki/Help:Wikitext Wikitext Help]

== Reducing the size of the VM image ==
Open a terminal in the VM and type

sudo dd if=/dev/zero | pv | sudo dd of=/bigemptyfile bs=4096k
sudo rm -rf /bigemptyfile

== Sandbox (for testing) ==

[[Test images]]

Developers Corner

2022-05-31T15:13:55Z

Conor:

ICTP2020

2020-01-23T14:34:35Z

Conor: /* Monday 27 Jan HANDS-ON 1 */

Plan for the ICTP 2020 school tutorials

=== Tutorial files ===
All tutorials require '''download''' of the following pre-prepared Yambo databases or DFT input files: 
DAY 1-2-3. hBN-3D and hBN-2D
[http://www.yambo-code.org/educational/tutorials/files/hBN.tar.gz hBN.tar.gz] (237 MB)
[http://www.yambo-code.org/educational/tutorials/files/hBN-2D.tar.gz hBN-2D.tar.gz] (8.6 MB) 
DAY 3. hBN-2D with higher convergence parameters for parallel tutorials:
[http://www.yambo-code.org/educational/tutorials/files/hBN-2D-para.tar.gz hBN-2D-para.tar.gz] (137 MB) 
DAY 4. hBN-2D and AlAs for real time simulations
[http://www.yambo-code.org/educational/tutorials/files/hBN-2D-RT.tar.gz hBN-2D-RT.tar.gz] (10 MB)
[http://www.yambo-code.org/educational/tutorials/files/AlAs.tar.gz AlAs.tar.gz] (10 MB)

After downloading the tar.gz files just unpack them in '''the same folder''':
$ tar -xcvf hBN-2D.tar
$ tar -xcvf hBN.tar
$ ls
YAMBO_TUTORIALS
$ ls YAMBO_TUTORIALS
hBN-2D hBN

=== Monday 27 Jan HANDS-ON 1 ===
'''14:40 - 17:00 From the DFT ground state to the complete setup of a Many Body calculation using Yambo''' Davide Sangalli (CNR-ISM, Italy), Pedro Melo (University of Liege, Belgium)

* [[First steps: walk through from DFT to RPA (standalone)]]

=== Tuesday 28 Jan HANDS-ON 2 ===

'''14:00 - 18:00 A complete tour through GW simulation in a complex material (from the blackboard to numerical computation: convergence, algorithms, parallel usage)''' Daniele Varsano (CNR-NANO, Italy), Andrea Ferretti (CNR-NANO, Italy)

* [[How to obtain the quasi-particle band structure of a bulk material: h-BN]]

'''16:30 - 18:00 A complete tour through GW simulation in a complex material (from the blackboard to the computer settings: convergence, algorithms, parallel usage) (continued)''' Daniele Varsano (CNR-NANO, Italy), Andrea Ferretti (CNR-NANO, Italy)

* [[GW_parallel_strategies|Parallel GW]]: strategies for running Yambo in parallel
* [[Pushing_convergence_in_parallel|GW convergence]]: use Yambo in parallel to converge a GW calculation for a layer of hBN (hBN-2D)

Link to old CECAM specific cases: [[GW_parallel_strategies_CECAM]] and [[Pushing_convergence_in_parallel_CECAM]]

=== Wednesday 29 Jan HANDS-ON 3 ===

'''14:00 - 16:00 A guided tour through calculations of spectroscopic properties using the BSE approach''' Daniele Varsano (CNR-NANO, Italy), Maurizia Palummo (University of Rome Tor Vergata, Italy)

* [[How to obtain an optical spectrum|Calculating optical spectra including excitonic effects: a step-by-step guide]]
* [[How to choose the input parameters|Obtaining a converged optical spectrum]]

'''16:30 - 18:00 Many-body effects in 2D materials: convergences, exciton characterizations''' Maurizia Palummo (University of Rome Tor Vergata, Italy), Attaccalite Claudio (CNRS, CINAM, Aix-Marseille Univ., France)

* [[How to treat low dimensional systems|Many-body effects in low-dimensional systems: numerical issues and remedies]]
* [[How to analyse excitons|Analysis of excitonic spectra in a 2D material]]

=== Thursday 30 Jan HANDS-ON 4 ===

'''14:00 - 15:00 Real-time approach and Calculation of linear response functions and optical properties''' Claudio Attaccalite (CNRS, CINAM, Aix-Marseille Univ., France), Davide Sangalli (CNR-ISM, Italy)

* [[Linear response from real time simulations]]

'''15:30 - 18:00 Real time approach and Calculation of non linear properties (second harmonic generation) ''' Claudio Attaccalite (CNRS, CINAM, Aix-Marseille Univ., France), Davide Sangalli (CNR-ISM, Italy)

* [[Real time approach to non-linear response]]
* [[Correlation effects in the non-linear response]]

=== Friday 31 Jan HANDS-ON 5 ===

'''09:00 - 10:30 Python scripting tools for accelerated GW convergence ''' Fulvio Paleari (CNR-ISM, Italy), Alejandro Molina-Sanchez (IINL, Portugal)

* [[First steps in Yambopy]]
* [[GW tutorial. Convergence and approximations (BN)]]

'''10:30 - 11:30 Python scripting tools for BSE convergence and analysis ''' Fulvio Paleari (CNR-ISM, Italy), Pedro Melo

* [[Bethe-Salpeter equation tutorial. Optical absorption (BN)]]

First steps: walk through from DFT to RPA (standalone)

2020-01-23T14:30:44Z

Conor:

In this tutorial you will learn how to calculate optical spectra using Yambo, starting from a DFT calculation and ending with a look at local field effects in the optical response.

== System characteristics ==
We will use a 3D system (bulk hBN) and a 2D system (hBN sheet).

[[File:HBN-bulk-3x3-annotated.png|x200px|Atomic structure of bulk hBN]]
[[File:HBN2.png|x200px|Atomic structure of 2D hBN]]

'''Hexagonal boron nitride - hBN''':
* HCP lattice, ABAB stacking
* Four atoms per cell, B and N (16 electrons)
* Lattice constants: ''a'' = 4.716 [a.u.], ''c/a'' = 2.582
* Plane wave cutoff 40 Ry (~1500 RL vectors in wavefunctions)
* SCF run: shifted ''6x6x2'' grid (12 k-points) with 8 bands
* Non-SCF run: gamma-centred ''6x6x2'' (14 k-points) grid with 100 bands

=== Prerequisites ===
'''You will need''':
* PWSCF input files and pseudopotentials for hBN bulk
* <code>pw.x</code> executable, version 5.0 or later
* <code>p2y</code> and <code>yambo</code> executables
* <code>gnuplot</code> for plotting spectra

==Download the Files==

Download and unpack the [http://www.yambo-code.org/educational/tutorials/files/hBN.tar.gz hBN.tar.gz] and [http://www.yambo-code.org/educational/tutorials/files/hBN-2D.tar.gz hBN-2D.tar.gz] files.
After downloading the tar.gz files just unpack them in the '''YAMBO_TUTORIALS''' folder. For example
$ mkdir YAMBO_TUTORIALS
$ mv hBN.tar.gz YAMBO_TUTORIALS
$ cd YAMBO_TUTORIALS
$ tar -xvfz hBN.tar.gz
$ ls YAMBO_TUTORIALS
hBN

(Advanced users can download and install all tutorial files using git. See the main [[Tutorials#Files|Tutorial Files]] page.)

==DFT calculation of bulk hBN and conversion to Yambo==

In this module you will learn how to generate the Yambo ''SAVE'' folder for bulk hBN starting from a PWscf calculation.

=== DFT calculations ===
$ cd YAMBO_TUTORIALS/hBN/PWSCF
$ ls
Inputs Pseudos PostProcessing References
hBN_scf.in hBN_nscf.in hBN_scf_plot_bands.in hBN_nscf_plot_bands.in

First run the SCF calculation to generate the ground-state charge density, occupations, Fermi level, and so on:
$ pw.x < hBN_scf.in > hBN_scf.out
Inspection of the output shows that the valence band maximum lies at 5.06eV.

Next run a non-SCF calculation to generate a set of Kohn-Sham eigenvalues and eigenvectors for both occupied and unoccupied states (100 bands):
$ pw.x < hBN_nscf.in > hBN_nscf.out ''(serial run, ~1 min) OR''
$ mpirun -np 2 pw.x < hBN_nscf.in > hBN_nscf.out ''(parallel run, 40s)''
Here we use a ''6x6x2'' grid giving 14 k-points, but denser grids should be used for checking convergence of Yambo runs.

Note the presence of the following flags in the input file:
wf_collect=.true.
force_symmorphic=.true.
diago_thr_init=5.0e-6,
diago_full_acc=.true.
which are needed for generating the Yambo databases accurately. Full explanations of these variables are given on the [http://www.quantum-espresso.org/wp-content/uploads/Doc/INPUT_PW.html quantum-ESPRESSO input variables page].

After these two runs, you should have a ''hBN.save'' directory:
$ ls hBN.save
data-file.xml charge-density.dat gvectors.dat B.pz-vbc.UPF N.pz-vbc.UPF
K00001 K00002 .... K00035 K00036

=== Conversion to Yambo format ===
The PWscf ''bBN.save'' output is converted to the Yambo format using the <code>p2y</code> executable (pwscf to yambo), found in the yambo ''bin'' directory.
Enter ''hBN.save'' and launch <code>p2y</code>:

$ cd hBN.save
$ p2y
...
<---> DBs path set to .
<---> Index file set to data-file.xml
<---> Header/K-points/Energies... done
...
<---> == DB1 (Gvecs and more) ...
<---> ... Database done
<---> == DB2 (wavefunctions) ... done ==
<---> == DB3 (PseudoPotential) ... done ==
<---> == P2Y completed ==

This output repeats some information about the system and generates a ''SAVE'' directory:
$ ls SAVE
ns.db1 ns.wf ns.kb_pp_pwscf
ns.wf_fragments_1_1 ...
ns.kb_pp_pwscf_fragment_1 ...
These files, with an ''n'' prefix, indicate that they are in netCDF format, and thus not human readable. However, they are perfectly transferable across different architectures. You can check that the databases contain the information you expect by launching Yambo using the <code>-D</code> option:

$ yambo -D
[RD./SAVE//ns.db1]------------------------------------------
Bands : 100
K-points : 14
G-vectors [RL space]: 8029
Components [wavefunctions]: 1016
...
[RD./SAVE//ns.wf]-------------------------------------------
Fragmentation :yes
...
[RD./SAVE//ns.kb_pp_pwscf]----------------------------------
Fragmentation :yes
- S/N 006626 -------------------------- v.04.01.02 r.00000 -

In practice we suggest to move the ''SAVE'' folder into a new clean folder.

In this tutorial however, we ask instead that you continue using a ''SAVE'' folder that we prepared previously:
$ cd ../../YAMBO
$ ls
SAVE

==Initialization of Yambo databases==
Use the ''SAVE'' folders that are already provided, rather than any ones you may have generated previously.

Every Yambo run '''must''' start with this step. Go to the folder ''containing'' the hBN-bulk <code>SAVE</code> directory:
$ cd YAMBO_TUTORIALS/hBN/YAMBO
$ ls
SAVE
and simply launch the code
$ yambo
This will run the initialization (setup) ''runlevel''.

'''TIP''': do not run yambo from ''inside'' the <code>SAVE</code> folder! In fact, if you ever see the message:
yambo: cannot access CORE database (SAVE/*db1 and/or SAVE/*wf)
it usually means you are trying to launch Yambo '''from the wrong place'''.

Three new elements will appear:
===Run-time output===
This is typically written to standard output (on screen) and tracks the progress of the run in real time:
<---> [01] CPU structure, Files & I/O Directories
<---> [02] CORE Variables Setup
<---> [02.01] Unit cells
<---> [02.02] Symmetries
<---> [02.03] RL shells
<---> Shells finder |########################################| [100%] --(E) --(X)
<---> [02.04] K-grid lattice
<---> [02.05] Energies [ev] & Occupations
<---> [03] Transferred momenta grid
<---> BZ -> IBZ reduction |########################################| [100%] --(E) --(X)
<---> X indexes |########################################| [100%] --(E) --(X)
<---> SE indexes |########################################| [100%] --(E) --(X)
<---> [04] External corrections
<---> [05] Game Over & Game summary
Specific runlevels are indicated with numeric labels like [02.02]. 
The hashes (#) indicate progress of the run in Wall Clock time, indicating the elapsed (E) and expected (X) time to complete a runlevel, and the percentage of the task complete.

If Yambo is launched using a script, or as a background process, or in parallel, this output will appear in a log file prefixed by the letter ''l'', in this case as ''l_setup''.

If this log file already exists from a previous run, it will not be overwritten. Instead, a new file will be created with an incrementing numerical label, e.g. ''l_setup_01, l_setup_02'', etc. '''This applies to all files created by Yambo'''.

In the case of parallel runs, CPU-dependent log files will appear inside a ''LOG'' folder, e.g.
$ mpirun -np 4 yambo
$ ls LOG
l_setup_CPU_1 l_setup_CPU_2 l_setup_CPU_3 l_setup_CPU_4
This behaviour can be controlled at runtime - see the Parallel tutorial for details.

===New core databases===
New databases appear in the ''SAVE'' folder:
$ ls SAVE
ns.db1 ns.wf ns.kb_pp_pwscf '''ndb.gops ndb.kindx'''
ns.wf_fragments_1_1 ...
ns.kb_pp_pwscf_fragment_1 ...
These contain information about the ''G''-vector shells and ''k/q''-point meshes as defined by the DFT calculation.

In general: a database called ''n'''s'''.xxx'' is a ''static'' database, generated once by <code>p2y</code>, while databases called ''n'''db'''.xxx'' are ''dynamically'' generated while you use <code>yambo</code>.

'''TIP''': if you launch <code>yambo</code>, but it does not seem to do anything, check that these files are present.

===Report file===
A report file ''r_setup'' is generated in the run directory.
This mostly reports information about the ground state system as defined by the DFT run, but also adds information about the band gaps, occupations, shells of G-vectors, IBZ/BZ grids, the CPU structure (for parallel runs), and so on. Some points of note:

[02.03] RL shells
=================
Shells, format: [S#] G_RL(mHa)
[S453]:8029(0.7982E+5) [S452]:8005(0.7982E+5) [S451]:7981(0.7982E+5) [S450]:7957(0.7942E+5)
...
[S4]:11( 1183.) [S3]:5( 532.5123) [S2]:3( 133.1281) [S1]:1( 0.000000)

This reports the set of closed reciprocal lattice (RL) shells defined internally that contain G-vectors with the same modulus.
The highest number of RL vectors we can use is 8029. Yambo will always redefine any input variable in RL units to the nearest closed shell.

[02.05] Energies [ev] & Occupations
===================================
Fermi Level [ev]: 5.112805
VBM / CBm [ev]: 0.000000 3.876293
Electronic Temp. [ev K]: 0.00 0.00
Bosonic Temp. [ev K]: 0.00 0.00
El. density [cm-3]: 0.460E+24
States summary : Full Metallic Empty
0001-0008 0009-0100
Indirect Gaps [ev]: 3.876293 7.278081
Direct Gaps [ev]: 4.28829 11.35409
X BZ K-points : 72

Yambo recalculates again the Fermi level (close to the value of 5.06 noted in the PWscf SCF calculation). From here on, however, the Fermi level is set to zero, and other eigenvalues are shifted accordingly. The system is insulating (8 filled, 92 empty) with an indirect band gap of 3.87 eV. The minimum and maximum direct and indirect gaps are indicated. There are 72 k-points in the full BZ, generated using symmetry from the 14 k-points in our user-defined grid.

'''TIP''': You should inspect the report file after ''every'' run for errors and warnings.

===2D hBN===
Simply repeat the steps above. Go to the folder ''containing'' the hBN-sheet ''SAVE'' directory and launch <code>yambo</code>:
$ cd TUTORIALS/hBN-2D/YAMBO
$ ls
SAVE
$ yambo
Again, inspect the ''r_setup'' file, output logs, and verify that ''ndb.gops'' and ''ndb.kpts'' have been created inside the SAVE folder.

You are now ready to use Yambo!

==Yambo's command line interface==
Yambo uses a command line interface to select tasks, generate input files, and control the runtime behaviour.

In this module you will learn how to select tasks, generate and modify input files, and control the runtime behaviour by using Yambo's command line interface.

Command line options are divided into '''uppercase''' and '''lowercase''' options:
* Lowercase: select tasks, generate input files, and (by default) launch a file editor
* Uppercase: modify Yambo's default settings, at run time and when generating input files
Lowercase and uppercase options can be used together.

=== Input file generator ===
First, move to the appropriate folder and initialize the Yambo databases if you haven't already done so.
$ cd YAMBO_TUTORIALS/hBN/YAMBO
$ yambo ''(initialize)''

Yambo generates its own input files: you just tell the code what you want to calculate by launching Yambo along with one or more '''lowercase''' options.

====Allowed options====
To see the list of runlevels and options, run <code>yambo -h</code> or better,
$ yambo -H
This is yambo 4.4.0 rev.148
A shiny pot of fun and happiness [C.D.Hogan]
-h :Short Help
-H :Long Help
-J <opt> :Job string identifier
-V <opt> :Input file verbosity[opt=RL,kpt,sc,qp,io,gen,resp,all,par]
-F <opt> :Input file
-I <opt> :Core I/O directory
-O <opt> :Additional I/O directory
-C <opt> :Communications I/O directory
-D :DataBases properties
-W <opt> :Wall Time limitation (1d2h30m format)
-Q :Don't launch the text editor
-E <opt> :Environment Parallel Variables file
-M :Switch-off MPI support (serial run)
-N :Switch-off OpenMP support (single thread run)
-i :Initialization
-o <opt> :Optics [opt=(c)hi is (G)-space / (b)se is (eh)-space ]
-k <opt> :Kernel [opt=hartree/alda/lrc/hf/sex](hf/sex only eh-space; lrc only G-space)
-y <opt> :BSE solver [opt=h/d/s/(p/f)i](h)aydock/(d)iagonalization/(i)nversion
-r :Coulomb potential
-x :Hartree-Fock Self-energy and local XC
-d :Dynamical Inverse Dielectric Matrix
-b :Static Inverse Dielectric Matrix
-p <opt> :GW approximations [opt=(p)PA/(c)HOSEX]
-g <opt> :Dyson Equation solver[opt=(n)ewton/(s)ecant/(g)reen]
-l :GoWo Quasiparticle lifetimes
-a :ACFDT Total Energy
-s :ScaLapacK test

Any time you launch Yambo with a lowercase option, Yambo will generate the appropriate input file (default name: ''yambo.in'') and launch the <code>vi</code> editor.

Editor choice can be changed at configure; alternatively you can use the <code>-Q</code> run time option to skip the automatic editing (do this if you are not familiar with <code>vi</code>!):
$ yambo -x -Q
yambo: input file yambo.in created
$ emacs yambo.in ''or your favourite editing tool''

====Combining options====
Multiple options can be used together to activate various tasks or runlevels (in some cases this is actually a necessity).
For instance, to generate an input file for optical spectra including local field effects (Hartree approximation), do (and then exit)
$ yambo -o c -k hartree ''which switches on:''
optics # [R OPT] Optics
chi # [R CHI] Dyson equation for Chi.
Chimod= "Hartree" # [X] IP/Hartree/ALDA/LRC/BSfxc
To perform a Hartree-Fock and GW calculation using a plasmon-pole approximation, do (and then exit):
$ yambo -x -g n -p p ''which switches on:''
HF_and_locXC # [R XX] Hartree-Fock Self-energy and Vxc
gw0 # [R GW] GoWo Quasiparticle energy levels
ppa # [R Xp] Plasmon Pole Approximation
em1d # [R Xd] Dynamical Inverse Dielectric Matrix
Each runlevel activates its own list of variables and flags.

===Changing input parameters ===
Yambo reads various parameters from existing database files and/or input files and uses them to suggest values or ranges.
Let's illustrate this by generating the input file for a Hartree-Fock calculation.

$ yambo -x
Inside the generated input file you should find:
[[Variables#EXXRLvcs|EXXRLvcs]] = 3187 RL # [XX] Exchange RL components
%[[Variables#QPkrange|QPkrange]] # [GW] QP generalized Kpoint/Band indices
1| 14| 1|100|
%
The <code>[[Variables#QPkrange|QPkrange]]</code> variable (follow the link for a "detailed" explanation for any variable) suggests a range of k-points (1 to 14) and bands (1 to 100) based on what it finds in the core database ''SAVE/ns.db1'', i.e. as defined by the DFT code. 
Leave that variable alone, and instead modify the previous variable to <code>EXXRLvcs= 1000 RL</code>

Save the file, and now generate the input a second time with <code>yambo -x</code>. You will see:
[[Variables#EXXRLvcs|EXXRLvcs]]= 1009 RL
This indicates that Yambo has read the new input value (1000 G-vectors), checked the database of G-vector shells ''(SAVE/ndb.gops)'',
and changed the input value to one that fits a completely closed shell.

Last, note that Yambo variables can be expressed in different '''units'''. In this case, <code>RL</code> can be replaced by an energy unit like Ry, eV, Ha, etc. Energy units are generally better as they are independent of the cell size. Technical information is available on the [[Variables]] page.

The input file generator of Yambo is thus an ''intelligent'' parser, which interacts with the user and the existing databases. For this reason we recommend that you always use Yambo to generate the input files, rather than making them yourself.

===Uppercase options===
Uppercase options modify some of the code's default settings. They can be used when launching the code but also when generating input files.

====Allowed options====
To see the list of options, again do:
$ yambo -H
Tool: yambo 4.1.2 rev.14024
Description: A shiny pot of fun and happiness [C.D.Hogan]
-J <opt> :Job string identifier
-V <opt> :Input file verbosity
[opt=RL,kpt,sc,qp,io,gen,resp,all,par]
-F <opt> :Input file
-I <opt> :Core I/O directory
-O <opt> :Additional I/O directory
-C <opt> :Communications I/O directory
-D :DataBases properties
-W <opt> :Wall Time limitation (1d2h30m format)
-Q :Don't launch the text editor
-M :Switch-off MPI support (serial run)
-N :Switch-off OpenMP support (single thread run)
[Lower case options]

Command line options are extremely important to master if you want to use yambo productively.
Often, the meaning is clear from the help menu:
$ yambo -F yambo.in_HF -x ''Make a Hartree -Fock input file called yambo.in_HF''
$ yambo -D ''Summarize the content of the databases in the SAVE folder''
$ yambo -I ../ ''Run the code, using a SAVE folder in a directory one level up''
$ yambo -C MyTest ''Run the code, putting all report, log, plot files inside a folder MyTest''

Other options deserve a closer look.
===Verbosity===
Yambo uses ''many'' input variables, many of which can be left at their default values. To keep input files short and manageable, only a few variables appear by default in the inout file. More advanced variables can be switched on by using the <code>-V</code> verbosity option. These are grouped according to the type of variable. For instance, <code>-V RL</code> switches on variables related to G vector summations, and <code>-V io</code> switches on options related to I/O control. Try:

$ yambo -o c -V RL ''switches on:''
FFTGvecs= 3951 RL # [FFT] Plane-waves

$ yambo -o c -V io ''switches on:''
StdoHash= 40 # [IO] Live-timing Hashes
DBsIOoff= "none" # [IO] Space-separated list of DB with NO I/O. DB= ...
DBsFRAGpm= "none" # [IO] Space-separated list of +DB to be FRAG and ...
#WFbuffIO # [IO] Wave-functions buffered I/O

Unfortunately, -V options must be invoked and changed ''one at a time''. When you are more expert, you may go straight to <code>-V all</code>, which turns on all possible variables. However note that <code>yambo -o c -V all</code> adds an extra 30 variables to the input file, which can be confusing: use it with care.

===Job script label===
The best way to keep track of different runs using different parameters is through the <code>-J</code> flag. This inserts a label in all output and report files, and creates a new folder containing any new databases (i.e. they are not written in the core ''SAVE'' folder). Try:
$ yambo -J 1Ry -V RL -x ''and modify to''
FFTGvecs = 1 Ry
EXXGvecs = 1 Ry
$ yambo -J 1Ry ''Run the code''
$ ls
yambo.in SAVE
o-1Ry.hf r-1Ry_HF_and_locXC 1Ry 1Ry/ndb.HF_and_locXC
This is extremely useful when running convergence tests, trying out different parameters, etc.

''Exercise'': use <code>yambo</code> to report the properties of all database files (including ''ndb.HF_and_locXC'')

==Optical absorption in hBN: independent particle approximation ==

=== Background ===
The dielectric function in the long-wavelength limit, at the independent particle level (RPA without local fields), is essentially given by the following:
[[File:CH1.png|none|x50px|Yambo tutorial image]]
In practice, Yambo does not use this expression directly but solves the Dyson equation for the susceptibility ''X'', which is described in the [[Local fields]] module.

=== Choosing input parameters ===
Enter the folder for bulk hBN that contains the ''SAVE'' directory, run the initialization and generate the input file. From <code>yambo -H</code> you should understand that the correct option is <code>yambo -o c</code>. Let's add some command line options:

$ cd YAMBO_TUTORIALS/hBN/YAMBO
$ yambo ''(initialization)''
$ yambo -F yambo.in_IP -o c
This corresponds to optical properties in G-space at the independent particle level: in the input file this is indicated by (<code>Chimod= "IP"</code>).

===Optics runlevel===
For optical properties we are interested just in the long-wavelength limit ''q = 0''. This always corresponds to the ''first'' ''q''-point in the set of possible ''q=k-k' ''-points.
Change the following variables in the input file to:
% [[Variables#QpntsRX|QpntsRXd]]
1 | '''1''' | # [Xd] Transferred momenta
%
[[Variables#ETStpsX|ETStpsXd]]= '''1001''' # [Xd] Total Energy steps
in order to select just the first ''q''. The last variable ensures we generate a smooth spectrum.
Save the input file and launch the code, keeping the command line options as before (i.e., just remove the lower case options):
$ yambo -F yambo.in_IP -J Full
...
<---> [05] Optics
<---> [LA] SERIAL linear algebra
<---> [DIP] Checking dipoles header
<---> [x,Vnl] computed using 4 projectors
<---> [M 0.017 Gb] Alloc WF ( 0.016)
<---> [WF] Performing Wave-Functions I/O from ./SAVE
<01s> Dipoles: P and iR (T): |########################################| [100%] 01s(E) 01s(X)
<01s> [M 0.001 Gb] Free WF ( 0.016)
<01s> [DIP] Writing dipoles header
<01s> [X-CG] R(p) Tot o/o(of R) : 5501 52992 100
<01s> Xo@q[1] |########################################| [100%] --(E) --(X)
<01s> [06] Game Over & Game summary

$ ls
Full SAVE yambo.in_IP r_setup
o-Full.eel_q1_ip o-Full.eps_q1_ip r-Full_optics_chi
Let's take a moment to understand what Yambo has done inside the Optics runlevel:
# Compute the ''[x,Vnl]'' term
# Read the wavefunctions from disc [WF]
# Compute the ''dipoles'', i.e. matrix elements of '''p'''
# Write the dipoles to disk as ''Full/ndb.dip*'' databases. This you can see in the report file:
$ grep -A20 "WR" r-Full_optics_chi
[WR./Full//ndb.dip_iR_and_P]
Brillouin Zone Q/K grids (IBZ/BZ): 14 72 14 72
RL vectors (WF): 1491
Electronic Temperature [K]: 0.0000000
Bosonic Temperature [K]: 0.0000000
X band range : 1 100
RL vectors in the sum : 1491
[r,Vnl] included :yes
...
<ol start="5">
<li>Finally, Yambo computes the non-interacting susceptibility ''X0'' for this ''q'', and writes the dielectric function inside the ''o-Full.eps_q1_ip'' file for plotting</li>
</ol>

===Energy cut off===

Before plotting the output, let's change a few more variables. The previous calculation used ''all'' the G-vectors in expanding the wavefunctions, up to 1491 (~1016 components). This corresponds roughly to the cut off energy of 40Ry we used in the DFT calculation. Generally, however, we can use a smaller value. We use the verbosity to switch on this variable, and a new ''-J'' flag to avoid reading the previous database:
$ yambo -F yambo.in_IP '''-J 6Ry -V RL''' -o c
Change the '''value''' of <code>[[Variables#FFTGvecs|FFTGvecs]]</code> and also its '''unit''' from <code>RL</code> (number of G-vectors) to <code>Ry</code> (energy in Rydberg):
[[Variables#FFTGvecs|FFTGvecs]]= '''6''' '''Ry''' # [FFT] Plane-waves
Save the input file and launch the code again:
$ yambo -F yambo.in_IP '''-J 6Ry -V RL'''
and then plot the ''o-Full.eps_q1_ip'' and ''o-6Ry.eps_q1_ip'' files:
$ gnuplot
gnuplot> plot "o-Full.eps_q1_ip" w l,"o-6Ry.eps_q1_ip" w p

[[File:CH-hBN-6Ry.png|none|500px|Yambo tutorial image]]

Clearly there is very little difference between the two spectra. This highlights an important point in calculating excited state properties: generally, fewer G-vectors are needed than what are needed in DFT calculations. Regarding the spectrum itself, the first peak occurs at about 4.4eV. This is consistent with the minimum direct gap reported by Yambo: 4.28eV. The comparison with experiment (not shown) is very poor however.

If you make some mistake, and cannot reproduce this figure, you should check the value of <code>[[Variables#FFTGvecs|FFTGvecs]]</code> in the input file, delete the ''6Ry'' folder, and try again - taking care to plot the right file! (e.g. ''o-6Ry.eps_q1_ip_01'').

===q-direction===
Now let's select a different component of the dielectric tensor:
$ yambo -F yambo.in_IP -J 6Ry -V RL -o c
...
% [[Variables#LongDrXd|LongDrXd]]
'''0.000000''' | 0.000000 | '''1.000000''' | # [Xd] [cc] Electric Field
%
...
$ yambo -F yambo.in_IP -J 6Ry -V RL
This time yambo reads from the ''6Ry'' folder, so it does not need to compute the dipole matrix elements again, and the calculation is fast. Plotting gives:
$ gnuplot
gnuplot> plot "o-6Ry.eps_q1_ip" t "q || x-axis" w l,"o-6Ry.eps_q1_ip_01" t "q || c-axis" w l

[[File:CH-hBN-ac.png|none|500px|Yambo tutorial image]]
The absorption is suppressed in the stacking direction. As the interplanar spacing is increased, we would eventually arrive at the absorption of the BN sheet (see [[Local fields]] tutorial).

===Non-local commutator===
Last, we show the effect of switching off the non-local commutator term (see ''[Vnl,r]'' in the equation at the top of the page) due to the pseudopotential. As there is no option to do this inside yambo, you need to hide the database file. Change back to the ''q || (1 0 0)'' direction, and launch yambo with a different <code>-J</code> option:
$ mv SAVE/ns.kb_pp_pwscf SAVE/ns.kb_pp_pwscf_'''OFF'''
$ yambo -F yambo.in_IP -J '''6Ry_NoVnl''' -o c ''(change to q || 100)''
$ yambo -F yambo.in_IP -J 6Ry_NoVnl
Note the warning in the output:
<---> [WARNING] Missing non-local pseudopotential contribution
which also appears in the report file, and noted in the database as <code>[r,Vnl] included :no</code>. The difference is tiny:
[[File:CH-hBN-Vnl.png|none|500px|Yambo tutorial image]]

However, when your system is larger, with more projectors in the pseudopotential or more k-points (see the BSE tutorial), the inclusion of ''Vnl'' can make a huge difference in the computational load, so it's always worth checking to see if the terms are important in your system.

==Optical absorption in 2D BN: local field effects ==

=== Background ===
[[File:Yambo-handbook-v4.1.2-p-4.png|thumb|Cheatsheet on LFE|150px]]
The macroscopic dielectric function is obtained by including the so-called local field effects (LFE) in the calculation of the response function. Within the time-dependent DFT formalism this is achieved by solving the Dyson equation for the susceptibility ''X''. In reciprocal space this is given by:
[[File:Yambo-CH7.png|none|x50px|Yambo tutorial image]]
The microscopic dielectric function is related to ''X'' by:
[[File:Yambo-CH5.png|none|x30px|Yambo tutorial image]]
and the macroscopic dielectric function is obtained by taking the (0,0) component of the inverse microscopic one:
[[File:Yambo-CH6.png|none|x60px|Yambo tutorial image]]
Experimental observables like the optical absorption and the electron energy loss can be obtained from the macroscopic dielectric function:
[[File:Yambo-CH8.png|none|x60px|Yambo tutorial image]]

In the following we will neglect the ''f xc'' term: we perform the calculation at the RPA level and consider just the Hartree term (from ''vG'') in the kernel. If we also neglect the Hartree term, we arrive back at the independent particle approximation, since there is no kernel and ''X = X0''.

=== Choosing input parameters ===
Enter the folder for 2D hBN that contains the SAVE directory, and generate the input file. From <code>yambo -H</code> you should understand that the correct option is <code>yambo -o c -k hartree</code>. Let's start by running the calculation for light polarization ''q'' in the plane of the BN sheet:
$ cd YAMBO_TUTORIALS/hBN-2D/YAMBO
$ yambo ''(Initialization)''
$ yambo -F yambo.in_RPA -V RL -J q100 -o c -k hartree
We thus use a new input file ''yambo.in_RPA'', switch on the <code>FFTGvecs</code> variable, and label all outputs/databases with a ''q100'' tag. Make sure to set/modify all of the following variables to:
[[Variables#FFTGvecs|FFTGvecs]]= '''6 Ry''' # [FFT] Plane-waves
[[Variables#Chimod|Chimod]]= "Hartree" # [X] IP/Hartree/ALDA/LRC/BSfxc
[[Variables#NGsBlkXd|NGsBlkXd]]= ''' 3 Ry''' # [Xd] Response block size
% [[Variables#QpntsRXd|QpntsRXd]]
1 | 1 | # [Xd] Transferred momenta
%
% [[Variables#EnRngeXd|EnRngeXd]]
0.00000 | 20.00000 | eV # [Xd] Energy range
%
% [[Variables#DmRngeXd|DmRngeXd]]
0.200000 | 0.200000 | eV # [Xd] Damping range
%
[[Variables#ETStpsXd|ETStpsXd]]= 2001 # [Xd] Total Energy steps
% [[Variables#LongDrXd|LongDrXd]]
1.000000 | 0.000000 | 0.000000 | # [Xd] [cc] Electric Field
%
In this input file, we have:
* A ''q'' parallel to the sheet
* A wider energy range than before, and more broadening
* Selected the Hartree kernel, and expanded G-vectors in the screening up to 3 Ry (about 85 G-vectors)

===LFEs in periodic direction===
Now let's run the code with this new input file (CECAM in serial: about 2mins; parallel 4 tasks: 50s)
$ yambo -F yambo.in_RPA -V RL -J q100
and let's compare the absorption with and without the local fields included. By inspecting the ''o-q100.eps_q1_inv_rpa_dyson'' file we find
that this information is given in the 2nd and 4th columns, respectively:
$ head -n30 o-q100.eps_q1_inv_rpa_dyson
# Absorption @ Q(1) [q->0 direction] : 1.0000000 0.0000000 0.0000000
# E/ev[1] EPS-Im[2] EPS-Re[3] EPSo-Im[4] EPSo-Re[5]
Plot the result:
$ gnuplot
gnuplot> plot "o-q100.eps_q1_inv_rpa_dyson" u 1:2 w l,"o-q100.eps_q1_inv_rpa_dyson" u 1:4 w l
[[File:CH-LFE4.png|none|600px|Yambo tutorial image]]
It is clear that there is little influence of local fields in this case. This is generally the case for semiconductors or materials with a smoothly varying electronic density. We have also shown the EELS spectrum (''o-q100.eel_q1_inv_rpa_dyson'') for comparison.

===LFEs in non-periodic direction===
Now let's switch to ''q'' perpendicular to the BN plane:
$ yambo -F yambo.in_RPA -V RL -o c -k hartree ''and set''
...
% [[Variables#LongDrXd|LongDrXd]]
0.000000 | 0.000000 | '''1.000000''' | # [Xd] [cc] Electric Field
%

You can try out the default parallel usage now, or run again in serial, i.e.
$ yambo -F yambo.in_RPA -V RL -J '''q001''' ''(serial)''
$ mpirun -np 4 yambo -F yambo.in_RPA -V RL -J '''q001''' & ''(parallel, MPI only, 4 tasks)''
As noted previously, the ''log'' files in parallel appear in the LOG folder, you can follow the execution with <code>tail -F LOG/l-q001_optics_chi_CPU_1</code> .

Plotting the output file:
$ gnuplot
gnuplot> plot "o-q001.eps_q1_inv_rpa_dyson" u 1:2 w l,"o-q001.eps_q1_inv_rpa_dyson" u 1:4 w l
[[File:CH-LFE6.png|none|600px|Yambo tutorial image]]
In this case, the absorption is strongly blueshifted with respect to the in-plane absorption. Furthermore, the influence of local fields is striking, and quenches the spectrum strongly. This is the well known depolarization effect. Local field effects are much stronger in the perpendicular direction because the charge inhomogeneity is dramatic. Many G-vectors are needed to account for the sharp change in the potential across the BN-vacuum interface.

===Absorption versus EELS===
In order to understand this further, we plot the electron energy loss spectrum for this component and compare with the absorption:
$ gnuplot
gnuplot > plot "o-q001.eps_q1_inv_rpa_dyson" w l,"o-q001.eel_q1_inv_rpa_dyson" w l
[[File:CH-LFE5.png|none|600px|Yambo tutorial image]]
The conclusion is that the absorption and EELS coincide for isolated systems.
To understand why this is, you need to consider the role of the ''macroscopic'' screening in the response function and the long-range part of the Coulomb potential.
See e.g.<ref>TDDFT from molecules to solids: The role of long‐range interactions, F. Sottile et al, International journal of quantum chemistry 102 (5), 684-701 (2005)</ref>

First steps: walk through from DFT to RPA (standalone)

2020-01-23T14:25:20Z

2020-01-08T17:03:01Z

Conor:

Plan for the ICTP 2020 school tutorials

=== Monday 27 Jan HANDS-ON 1 ===
'''14:40 - 17:00 From the DFT ground state to the complete setup of a Many Body calculation using Yambo''' Davide Sangalli (CNR-ISM, Italy), Pedro Melo (University of Liege, Belgium)
* [[First steps: a walk through from DFT to optical properties]]

=== Tuesday 28 Jan HANDS-ON 2 ===

'''14:00 - 18:00 A complete tour through GW simulation in a complex material (from the blackboard to numerical computation: convergence, algorithms, parallel usage)''' Daniele Varsano (CNR-NANO, Italy), Andrea Ferretti (CNR-NANO, Italy)

* [[How to obtain the quasi-particle band structure of a bulk material: h-BN]]

'''16:30 - 18:00 A complete tour through GW simulation in a complex material (from the blackboard to the computer settings: convergence, algorithms, parallel usage) (continued)''' Daniele Varsano (CNR-NANO, Italy), Andrea Ferretti (CNR-NANO, Italy)

* [[GW_parallel_strategies|Parallel GW]]: strategies for running Yambo in parallel
* [[Pushing_convergence_in_parallel|GW convergence (CECAM specific)]]: use Yambo in parallel to converge a GW calculation for a layer of hBN (hBN-2D)

Link to old CECAM specific cases: [[GW_parallel_strategies_CECAM]]

=== Wednesday 29 Jan HANDS-ON 3 ===

'''14:00 - 16:00 A guided tour through calculations of spectroscopic properties using the BSE approach''' Daniele Varsano (CNR-NANO, Italy), Maurizia Palummo (University of Rome Tor Vergata, Italy)

* [[How to obtain an optical spectrum|Calculating optical spectra including excitonic effects: a step-by-step guide]]
* [[How to choose the input parameters|Obtaining a converged optical spectrum]]

'''16:30 - 18:00 Many-body effects in 2D materials: convergences, spin orbit effect, exciton characterizations''' Maurizia Palummo (University of Rome Tor Vergata, Italy), Attaccalite Claudio (CNRS, CINAM, Aix-Marseille Univ., France)

* [[How to treat low dimensional systems|Many-body effects in low-dimensional systems: numerical issues and remedies]]
* [[How to analyse excitons|Analysis of excitonic spectra in a 2D material]]

=== Thursday 30 Jan HANDS-ON 4 ===

'''14:00 - 15:00 Real time approach and Calculation of linear response functions and optical properties''' Claudio Attaccalite (CNRS, CINAM, Aix-Marseille Univ., France), Davide Sangalli (CNR-ISM, Italy)

* [[Real time approach to linear response]]

'''15:30 - 18:00 Real time approach and Calculation of non linear properties (second harmonic generation) ''' Claudio Attaccalite (CNRS, CINAM, Aix-Marseille Univ., France), Davide Sangalli (CNR-ISM, Italy)

* [[Real time approach to non linear response]]

=== Friday 31 Jan HANDS-ON 5 ===

'''09:00 - 10:30 Python scripting tools for accelerated GW convergence ''' Fulvio Paleari (CNR-ISM, Italy), Alejandro Molina-Sanchez (IINL, Portugal)

* [[First steps in Yambopy]]
* [[GW tutorial. Convergence and approximations (BN)]]

'''10:30 - 11:30 Python scripting tools for BSE convergence and analysis ''' Fulvio Paleari (CNR-ISM, Italy), Alejandro Molina-Sanchez (IINL, Portugal)

* [[Bethe-Salpeter equation tutorial. Optical absorption (BN)]]

GW parallel strategies CECAM

2020-01-08T17:01:34Z

Conor:

'''This modules contains very general discussions of the parallel environment of Yambo. The commands are specific to the CECAM cluster. For other architectures, follow instead the [[GW_parallel_strategies]] tutorial or just replace the parallel queue instructions with simple MPI commands.'''

In this tutorial we will see how to setup the variables governing the parallel execution of yambo in order to perform efficient calculations in terms of both cpu time and memory to solution. As a test case we will consider the hBN 2D material. Because of its reduced dimensionality, GW calculations turns out to be very delicate. Beside the usual convergence studies with respect to k-points and sums-over-bands, in low dimensional systems a sensible amount of vacuum is required in order to treat the system as isolated, translating into a large number of plane-waves. As for other tutorials, it is important to stress that this tutorial it is meant to illustrate the functionality of the key variables and to run in reasonable time, so it has not the purpose to reach the desired accuracy to reproduce experimental results. Moreover please also note that scaling performance illustrated below may be significantly dependent on the underlying parallel architecture. Nevertheless, general considerations are tentatively drawn in discussing the results.

==Files and Tools==
Database and tools can be downloaded here:

*[http://www.yambo-code.org/tutorials/files/hBN-2D-para.tar.gz hBN-2D-para.tar.gz]
*[http://www.yambo-code.org/tutorials/files/parse_yambo.py parse_yambo.py]
*[http://www.yambo-code.org/tutorials/files/run.sh run.sh]

and also
*[http://www.yambo-code.org/tutorials/files/hBN.tar.gz hBN.tar.gz]
*[http://www.yambo-code.org/tutorials/files/hBN-2D.tar.gz hBN-2D.tar.gz]

== Getting familiar with yambo in parallel ==

If you are not inside bellatrix, please follow the instructions in the tutorial home.
If you are inside bellatrix and in the proper folder
[cecam.school01@bellatrix yambo_YOUR_NAME]$ pwd
/scratch/cecam.schoolXY/yambo_YOUR_NAME
you can proceed. First you need to obtain the appropriate tarball
$ cp /scratch/cecam.school/hBN-2D-para.tar.gz ./ (Notice that this time there is not XY!)
$ tar -zxvf hBN-2D-para.tar.gz
$ ls
YAMBO_TUTORIALS
$ cd YAMBO_TUTORIALS/hBN-2D-para/YAMBO

To run a calculation on bellatrix you need to go via the queue system as explained in the Tutorials home.
Under the YAMBO folder, together with the SAVE folder, you will see the run.sh script

$ ls
parse_yambo.py parse_qp.py run.sh SAVE

First run the initialization as usual.
Then you need to generate the input file for a GW run.
$ yambo -g n -p p -F yambo_gw.in
The new input file should look like the folllowing.
Please remove the lines you see below with a stroke and set the parameters to the same values as below
$ cat yambo_gw.in
#
#
# Y88b / e e e 888~~\ ,88~-_
# Y88b / d8b d8b d8b 888 | d888 \
# Y88b/ /Y88b d888bdY88b 888 _/ 88888 |
# Y8Y / Y88b / Y88Y Y888b 888 \ 88888 |
# Y /____Y88b / YY Y888b 888 | Y888 /
# / / Y88b / Y888b 888__/ `88_-~
#
#
# GPL Version 4.1.2 Revision 120
# MPI+OpenMP Build
# http://www.yambo-code.org
#
ppa # [R Xp] Plasmon Pole Approximation
gw0 # [R GW] GoWo Quasiparticle energy levels
HF_and_locXC # [R XX] Hartree-Fock Self-energy and Vxc
em1d # [R Xd] Dynamical Inverse Dielectric Matrix
<s>X_Threads= 32 # [OPENMP/X] Number of threads for response functions </s>
<s>DIP_Threads= 32 # [OPENMP/X] Number of threads for dipoles </s>
<s>SE_Threads= 32 # [OPENMP/GW] Number of threads for self-energy </s>
EXXRLvcs= 21817 RL # [XX] Exchange RL components
Chimod= "" # [X] IP/Hartree/ALDA/LRC/BSfxc
% BndsRnXp
1 | 200 | # [Xp] Polarization function bands
%
NGsBlkXp= 4 Ry # [Xp] Response block size
% LongDrXp
1.000000 | 0.000000 | 0.000000 | # [Xp] [cc] Electric Field
%
PPAPntXp= 27.21138 eV # [Xp] PPA imaginary energy
% GbndRnge
1 | 200 | # [GW] G[W] bands range
%
GDamping= 0.10000 eV # [GW] G[W] damping
dScStep= 0.10000 eV # [GW] Energy step to evaluate Z factors
DysSolver= "n" # [GW] Dyson Equation solver ("n","s","g")
%QPkrange # [GW] QP generalized Kpoint/Band indices
1| 1| 3| 6|
%

You can now open the submission script and the submission script
$ cat run.sh

#!/bin/bash
#SBATCH -N 1
#SBATCH -t 06:00:00
#SBATCH -J test
#SBATCH --reservation=cecam_course
#SBATCH --tasks-per-node=1

nodes=1
nthreads=1
ncpu=`echo $nodes $nthreads 1 | awk '{print $1*$3/$2}'`

module purge
module load intel/16.0.3
module load intelmpi/5.1.3
bindir=/home/cecam.school/bin/

export OMP_NUM_THREADS=$nthreads

label=MPI${ncpu}_OMP${nthreads}
jdir=run_${label}
cdir=run_${label}.out

filein0=yambo_gw.in
filein=yambo_gw_${label}.in

cp -f $filein0 $filein
cat >> $filein << EOF
X_all_q_CPU= "1 1 $ncpu 1" # [PARALLEL] CPUs for each role
X_all_q_ROLEs= "q k c v" # [PARALLEL] CPUs roles (q,k,c,v)
X_all_q_nCPU_LinAlg_INV= $ncpu # [PARALLEL] CPUs for Linear Algebra
X_Threads= 0 # [OPENMP/X] Number of threads for response functions
DIP_Threads= 0 # [OPENMP/X] Number of threads for dipoles
SE_CPU= " 1 1 $ncpu" # [PARALLEL] CPUs for each role
SE_ROLEs= "q qp b" # [PARALLEL] CPUs roles (q,qp,b)
SE_Threads= 0

EOF

echo "Running on $ncpu MPI, $nthreads OpenMP threads"
srun -n $ncpu -c $nthreads $bindir/yambo -F $filein -J $jdir -C $cdir

As soon as you are ready submit the job.

$ sbatch run.sh

Yambo calculates the GW-qp corrections running on 1 MPI process with a single thread.
As you can see, monitoring the log file produced by yambo, the run takes some time, although
we are using minimal parameters.

The status of the jobs can be controlled via:
$ squeue -u $USER # to inspect the status of jobs
#(hint: make a unix alias, if you like)
$ scancel <jobid> # to delete jobs in the queue

== Pure MPI scaling with default parallelization scheme ==

Meanwhile we can run the code in parallel. Let's use 16 MPI process, still with a single thread.
To this end modify the job.sh script changing
#SBATCH --tasks-per-node=16

ncpu=`echo $nodes $nthreads 16 | awk '{print $1*$3/$2}'`

or set ncpu by hand:
ncpu=XX (e.g. 1 to 16)

This time the code should be much faster.
Once the run is over try to run the simulation also on 2, 4, 8.
Each time please remember to change both the number of tasks per node and the number of CPUs.
At the end you can try to produce a scaling plot.

To analyze the data you can use the phyton "yambo_parse.py" script which is also provided.

You can use it running
$ ./parse_yambo.py run*/r-*
where $jobstring is the string you used for $jobname without the explicit number of MPI tasks.

You should obtain something like that (but with more columns)
# ncores MPI threads Dipoles Xo X Sgm_x Sgm_c WALL_TIME
1 1 1 1.00s 7m39.00s 0.00s 3.00s 1m46.00s 09m38s
2 2 1 2.00s 3m48.00s 0.00s 2.00s 53.00s 04m55s
4 4 1 0.00s 1m56.00s 0.00s 1.00s 27.00s 02m33s
8 8 1 0.00s 1m2.00s 0.00s 0.00s 14.00s 01m26s
16 16 1 0.00s 35.00s 0.00s 0.00s 7.00s 54s

Plot the execution time vs the number of MPI tasks
and check (do a log plot) how far you are from the ideal linear scaling.
Below a similar plot performed on Fermi
[[File:scaling_mpi_fermi.jpg|750px|center]]

'''tips:''' 
- not all runlevels scale in in the same way 
- you should never overload the available number of cores

== Pure OpenMP scaling ==

Next step is instead to check the OpenMP scaling.
Set back
#SBATCH --tasks-per-node=16
and now use
nthreads=16
ncpu=`echo $nodes $nthreads 16 | awk '{print $1*$3/$2}'`

In order to do pure OpenMP scaling the three numbers must be the same.
Notice that here the last 16 entering the definition of "ncpu" defines the total number of cores (or better threads, see tips below).
Since we are already using 16 threads, we cannot also split among MPI tasks, i.e. ncpu will result equal to 1.
Try setting nthreads to 16, 8, 4 and 2 and again to plot the execution time vs the number of threads using the pyton script.
Again you should be able to produce a plot similar to the following

[[File:scaling_omp_fermi.jpg|750px|center]]

'''tips:''' 
- OpenMP usually shares the memory among threads, but not always 
- you should never overload the available number of cores 
- in principle we could overload the cores setting more threads than the available total number of cores, since a single core allows multi-thread operations

== MPI vs OpenMP scaling ==

Which is scaling better ? MPI or OpenMP ?
How is the memory distributed ?

Now you can try running simualations with hybrid strategies.
Try for example setting:

#SBATCH --tasks-per-node=16

nthreads= 4
ncpu=`echo $nodes $nthreads 16 | awk '{print $1*$3/$2}'`

We can try to do scaling keeping the total number of tasks per node at 16.
Parsing the data we will obtain something similar to

# ncores MPI threads Dipoles Xo X Sgm_x Sgm_c WALL_TIME
16 1 16 1.00s 38.00s 0.00s 2.00s 33.00s 01m25s
16 2 8 2.00s 34.00s 0.00s 1.00s 16.00s 01m06s
16 4 4 0.00s 34.00s 0.00s 1.00s 11.00s 56s
16 8 2 0.00s 33.00s 0.00s 0.00s 9.00s 52s
16 16 1 0.00s 35.00s 0.00s 0.00s 7.00s 54s

As you can see here the total CPU time decreases more and more moving the parallelization from the the OpenMP to the MPI level.
Sigma_c in particular scales better. However the fastest run is not the one with 16 MPI task but the 8 2 configuration
since Xo is faster using an hybrid MPI-OpenMP scheme.

However CPU time is not the only parameter you need to check.
Also the total memory usage is important
If you compare for example the two extreme cases (you can use)
$ grep Gb run_MPI1_OMP*/l* | grep Alloc (use this for the case with only one MPI proc)
$ grep Gb run_MPI*_OMP*/LOG/l*_1 | grep Alloc (use this for the case with more than one MPI proc)

For the case
# ncores MPI threads
16 1 16
<01s> [M 0.119 Gb] Alloc WF ( 0.112)
<03s> [M 0.314 Gb] Alloc WF ( 0.306)
<46s> [M 0.074 Gb] Alloc WF ( 0.056)
<50s> [M 0.321 Gb] Alloc WF ( 0.306)
the numbers reported above refer to the total amount of memory use in the run.

For the case
# ncores MPI threads
16 16 1
<02s> P0001: [M 0.034 Gb] Alloc WF ( 0.026)
<43s> P0001: [M 0.037 Gb] Alloc WF ( 0.019)
<45s> P0001: [M 0.091 Gb] Alloc WF ( 0.076)
the numbers reported above refer to the total amount of memory per MPI task.
Multiplying by 16 you obtain an estimate of the total memory: 0.091*16=1.456 (0.076*16=1.216)
These last two numbers have to be compared with 0.321 (0.306)
As you can see yambo is distributing memory, since the single MPI task uses less memory than
the total one needed (you can even compare with the serial case). However it is not as efficient as
OpenMP in doing so.

Using an hybrid scheme you may also consider running yambo on mode than one node.
To run on two nodes for example you need to set
#SBATCH -N 2

nodes=2
accordingly you can now set
nthreads= 4
ncpu=`echo $nodes $nthreads 16 | awk '{print $1*$3/$2}'`
This time you will use 32 cores with (16 per node) 4 OpenMP threads and 2*16/4=8 MPI tasks.

'''tips:''' 
- in real life calculations running on n_cores > 100 it is a good idea to adopt an hybrid approach 
- with OpenMP you cannot exit the single node, with MPI you can

== Advanced: Comparing different parallelization schemes (optional) ==

Up to now we used the default parallelization scheme.
Yambo also allows you to tune the parameters which controls the parallelization scheme.
To this end you can open again the job.sh script and modify the section where the yambo input
variables are set

X_all_q_CPU= "1 1 $ncpu 1" # [PARALLEL] CPUs for each role
X_all_q_ROLEs= "q k c v" # [PARALLEL] CPUs roles (q,k,c,v)
#X_all_q_nCPU_LinAlg_INV= $ncpu # [PARALLEL] CPUs for Linear Algebra
X_Threads= 0 # [OPENMP/X] Number of threads for response functions
DIP_Threads= 0 # [OPENMP/X] Number of threads for dipoles
SE_CPU= "1 1 $ncpu" # [PARALLEL] CPUs for each role
SE_ROLEs= "q qp b" # [PARALLEL] CPUs roles (q,qp,b)
SE_Threads= 0

In particular "X_all_q_CPU" sets how the MPI Tasks are distributed in the calculation of the response function.
The possibilities are shown in the "X_all_q_ROLEs". The same holds for "SE_CPU" and "SE_ROLEs" which control
how MPI Tasks are distributed in the calculation of the response function.

Please try different parallelization schemes and check the performances of Yambo.
In doing so you should also change the jobname in the run.sh script
label=MPI${ncpu}_OMP${nthreads}_scheme1

Using the python script, you can then chenck how speed, memory and load balance between the CPUs are affected.
For more details see also the [[Using_Yambo_in_parallel|Parallel module]]

'''tips: '''
- the product of the numbers entering each variable (i.e. X_all_q_CPU and SE_CPU) times the number of threads should always match the total number of cores (unless you want to overload the cores taking advantage of multi-threads) 
- using the X_Threads and SE_Threads variables you can think about setting different Hybrid schemes in between the screening and the self-energy runlevel.
- memory better scales if you parallelize on bands (c v b) 
- parallelization on k-points performs similarly to parallelization on bands, but memory requires more memory 
- parallelization on q-points requires much less communication in between the MPI tasks. It maybe useful if you run on more than one node and the inter-node connection is slow

 
{| style="width:100%" border="1"
|style="width:15%; text-align:left"|Prev: [[Tutorials|Tutorials Home]]
|style="width:50%; text-align:center"|Now: [[Tutorials|Tutorials Home]] --> [[GW_parallel_strategies|GW Parallel]]
|style="width:35%; text-align:right"|Next: [[Pushing_convergence_in_parallel|GW Convergence]]
|-
|}

2019-11-07T10:13:24Z

Conor:

Here you will find a collection of lectures given by members of the Yambo team over the years. 
Click the ''Download'' link under an image to open the PDF in your browser.
==Theoretical background: MBPT==

[[File:ASESMA_2015_MBPT_1.png|200px]]
[[Media:ASESMA_2015_MBPT_1.pdf|Download]]