Quasi-particles of a 2D system - Revision history

Giacomo.sesti: /* Step 2: GW parallel strategies */

2025-05-20T13:46:45Z

Step 2: GW parallel strategies

← Older revision		Revision as of 13:46, 20 May 2025
Line 183:		Line 183:
	Differently to the approach used up to now, we will not work interactively but rather we will submit a job script as is commonly done when using Yambo on clusters. So, now exit the interactive mode and in the login node access the folder:		Differently to the approach used up to now, we will not work interactively but rather we will submit a job script as is commonly done when using Yambo on clusters. So, now exit the interactive mode and in the login node access the folder:

	cd $CINECA_SCRATCH/YAMBO_TUTORIALS/~~MoS2_2Dquasiparticle_tutorial~~/02_GW_parallel		cd $CINECA_SCRATCH/YAMBO_TUTORIALS/MoS2_2Dquasiparticle_tutorial_Modena2025/02_GW_parallel

	Here, inspect the input <code>gw.in</code>. You should see we compute a much larger number of quasi-particles:		Here, inspect the input <code>gw.in</code>. You should see we compute a much larger number of quasi-particles:

Giacomo.sesti: /* Step 2: GW parallel strategies */

2025-05-19T21:50:54Z

Step 2: GW parallel strategies

← Older revision		Revision as of 21:50, 19 May 2025
Line 278:		Line 278:
	On a cluster like leonardo, to activate the full potential of the machine, it is useful to activate OpenMP threads by modifying cpus-per-task in the submission file. The product of the number of OpenMP and MPI tasks is equal to the total number of CPUs.		On a cluster like leonardo, to activate the full potential of the machine, it is useful to activate OpenMP threads by modifying cpus-per-task in the submission file. The product of the number of OpenMP and MPI tasks is equal to the total number of CPUs.

	Therefore, we can distribute 8 cpus with 2 MPI tasks and then use 2 OpenMP threads:		Therefore, we can distribute 4 cpus with 2 MPI tasks and then use 2 OpenMP threads:

	#!/bin/bash		#!/bin/bash

Giacomo.sesti: /* Step 2: GW parallel strategies */

2025-05-19T21:49:56Z

Step 2: GW parallel strategies

← Older revision		Revision as of 21:49, 19 May 2025
Line 217:		Line 217:
	#!/bin/bash		#!/bin/bash
	#SBATCH --account=tra25_yambo # Charge resources used by this job to specified account		#SBATCH --account=tra25_yambo # Charge resources used by this job to specified account
	#SBATCH --time=00:10:00 # Set a limit on the total run time of the job allocation in hh:mm:ss		#SBATCH --time=00:04:00 # Set a limit on the total run time of the job allocation in hh:mm:ss
	#SBATCH --job-name=rutile # Specify a name for the job allocation		#SBATCH --job-name=rutile # Specify a name for the job allocation
	#SBATCH --partition=dcgp_usr_prod # Request a specific partition for the resource allocation		#SBATCH --partition=dcgp_usr_prod # Request a specific partition for the resource allocation

Giacomo.sesti: /* Step 2: GW parallel strategies */

2025-05-19T21:47:50Z

Step 2: GW parallel strategies

← Older revision		Revision as of 21:47, 19 May 2025
Line 282:		Line 282:
	#!/bin/bash		#!/bin/bash
	#SBATCH --nodes=1		#SBATCH --nodes=1
	#SBATCH --ntasks-per-node=4		#SBATCH --ntasks-per-node=2
	#SBATCH --cpus-per-task=2		#SBATCH --cpus-per-task=2

Giacomo.sesti: /* Step 2: GW parallel strategies */

2025-05-19T21:47:11Z

Step 2: GW parallel strategies

← Older revision		Revision as of 21:47, 19 May 2025
Line 225:		Line 225:
	#		#
	#SBATCH --nodes=1 # Number of nodes to be allocated for the job		#SBATCH --nodes=1 # Number of nodes to be allocated for the job
	#SBATCH --ntasks-per-node=~~NCPU~~ # Number of MPI tasks invoked per node		#SBATCH --ntasks-per-node=4 # Number of MPI tasks invoked per node
	#SBATCH --cpus-per-task=~~NOMP~~ # Number of OMP threads per task		#SBATCH --cpus-per-task=1 # Number of OMP threads per task

	# load yambo and dependencies		# load yambo and dependencies
Line 272:		Line 272:
	For simplicity you could just type		For simplicity you could just type

	tail -f ~~run_MPI8_OMP1~~.out/LOG/l-*_CPU_1		tail -f run_MPI4_OMP1.out/LOG/l-*_CPU_1

	to monitor the progress in the master thread (<code>Ctrl+C</code> to exit). As you can see, the run takes some time, even though we are using minimal parameters.		to monitor the progress in the master thread (<code>Ctrl+C</code> to exit). As you can see, the run takes some time, even though we are using minimal parameters.

	On a cluster like ~~m100~~, to activate the full potential of the machine, it is useful to activate OpenMP threads by modifying cpus-per-task in the submission file. The product of the number of OpenMP and MPI tasks is equal to the total number of CPUs.		On a cluster like leonardo, to activate the full potential of the machine, it is useful to activate OpenMP threads by modifying cpus-per-task in the submission file. The product of the number of OpenMP and MPI tasks is equal to the total number of CPUs.

	Therefore, we can distribute 8 cpus with 4 MPI tasks and then use 2 OpenMP threads:		Therefore, we can distribute 8 cpus with 2 MPI tasks and then use 2 OpenMP threads:

	#!/bin/bash		#!/bin/bash

Giacomo.sesti: /* Step 2: GW parallel strategies */

2025-05-19T21:45:49Z

Step 2: GW parallel strategies

Giacomo.sesti: /* Step 2: GW parallel strategies */

2025-05-19T21:42:43Z

Step 2: GW parallel strategies

← Older revision		Revision as of 21:42, 19 May 2025
Line 225:		Line 225:
	#		#
	#SBATCH --nodes=1 # Number of nodes to be allocated for the job		#SBATCH --nodes=1 # Number of nodes to be allocated for the job
	#SBATCH --ntasks-per-node=4 # Number of MPI tasks invoked per node		#SBATCH --ntasks-per-node=NCPU # Number of MPI tasks invoked per node
	#SBATCH --cpus-per-task=1 # Number of OMP threads per task		#SBATCH --cpus-per-task=NOMP # Number of OMP threads per task

	# load yambo and dependencies		# load yambo and dependencies

Giacomo.sesti: /* Step 2: GW parallel strategies */

2025-05-19T21:41:35Z

Step 2: GW parallel strategies

← Older revision		Revision as of 21:41, 19 May 2025
Line 233:		Line 233:
	module use /leonardo/pub/userexternal/nspallan/spack-0.22.2-06/modules		module use /leonardo/pub/userexternal/nspallan/spack-0.22.2-06/modules
	module load yambo/5.3.0--intel-oneapi-mpi--2021.12.1--oneapi--2024.1.0		module load yambo/5.3.0--intel-oneapi-mpi--2021.12.1--oneapi--2024.1.0
	~~export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}~~

	export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}		export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}

Giacomo.sesti: /* Step 2: GW parallel strategies */

2025-05-19T21:41:14Z

Step 2: GW parallel strategies

← Older revision		Revision as of 21:41, 19 May 2025
Line 216:		Line 216:

	#!/bin/bash		#!/bin/bash
	#SBATCH --~~nodes~~=1		#SBATCH --account=tra25_yambo # Charge resources used by this job to specified account
	#SBATCH --~~ntasks-per-node~~=2		#SBATCH --time=00:10:00 # Set a limit on the total run time of the job allocation in hh:mm:ss
	#SBATCH --~~cpus~~-~~per-task~~=1		#SBATCH --job-name=rutile # Specify a name for the job allocation
	#SBATCH --partition=~~m100_sys_test~~		#SBATCH --partition=dcgp_usr_prod # Request a specific partition for the resource allocation
	#SBATCH --~~qos~~=~~qos_test~~		#SBATCH --gres=tmpfs:10g # List of generic consumable resources
	#SBATCH --~~reservation~~=~~s_tra_yambo~~		#SBATCH --qos=normal # Quality of service
	#SBATCH --~~time~~=~~0:30:00~~		#SBATCH --reservation=s_tra_yambo # Reservation specific to this school
	#~~SBATCH --account=tra23_Yambo~~		#
	#SBATCH --~~mem~~=~~230000MB~~		#SBATCH --nodes=1 # Number of nodes to be allocated for the job
	#SBATCH --~~job~~-~~name~~=~~rutile~~		#SBATCH --ntasks-per-node=4 # Number of MPI tasks invoked per node
	#SBATCH --~~error=err.job~~-%j		#SBATCH --cpus-per-task=1 # Number of OMP threads per task
	~~#SBATCH~~ -~~-output~~=~~out.job-%j~~

	# load yambo and dependencies		# load yambo and dependencies
	module purge		module purge
	module load ~~hpc-sdk~~/~~2022~~--~~binary~~		module load profile/candidate
	module load ~~spectrum_mpi~~/10.4.0--~~binary~~		module use /leonardo/pub/userexternal/nspallan/spack-0.22.2-06/modules
	export ~~PATH~~=~~/m100_work/tra23_Yambo/softwares/YAMBO/5.2-cpu/bin:~~$~~PATH~~		module load yambo/5.3.0--intel-oneapi-mpi--2021.12.1--oneapi--2024.1.0
			export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}

	export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}		export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}

Giacomo.sesti at 21:39, 19 May 2025

2025-05-19T21:39:30Z

← Older revision		Revision as of 21:39, 19 May 2025
Line 305:		Line 305:

	<!-- ==Step 3: Running on GPU== hidden no gpu partition in Yambo 2025 school		<!-- ==Step 3: Running on GPU== hidden no gpu partition in Yambo 2025 school
	!
	~~!For this part of the tutorial, we will repeat the calculation of before, making use of gpus.~~

	~~#So now move to:~~		For this part of the tutorial, we will repeat the calculation of before, making use of gpus.

	~~# 02_GW_gpu~~		So now move to:

	#Here we are using yet another capability of yambo: running on GPUs instead of CPUs. This usually leads to extreme speedups in the #calculations. Fortunately, the m100 cluster also has some GPU nodes! So now let us have a look at the submission script:		02_GW_gpu

	~~# vim gpu_job~~.sh		Here we are using yet another capability of yambo: running on GPUs instead of CPUs. This usually leads to extreme speedups in the calculations. Fortunately, the m100 cluster also has some GPU nodes! So now let us have a look at the submission script:

	~~# #!/bin/bash~~		vim gpu_job.sh
	~~# #SBATCH --nodes=1~~
	~~# #SBATCH --ntasks-per-node=2~~
	~~# #SBATCH --cpus-per-task=1~~
	~~# #SBATCH --gres=gpu:4~~

	#~~each GPU~~ node ~~contains four GPUs (<code>~~--gres=gpu:4</code>). In yambo, each GPU corresponds to a single MPI task, therefore we have still #the total number of tasks equal to 8. OpenMP threading is allowed - but not too much, otherwise we lose efficiency. Here in fact, we are not #using OpenMP so </code>export OMP_NUM_THREADS=1</code>.		#!/bin/bash
			#SBATCH --nodes=1
			#SBATCH --ntasks-per-node=2
			#SBATCH --cpus-per-task=1
			#SBATCH --gres=gpu:4

	#In ~~addition~~, ~~you will see that~~ in ~~order to run on GPUs~~ we are ~~now~~ using ~~a different executable than before, obtained with a GPU-specific #compilation of the~~ code.		each GPU node contains four GPUs (<code>--gres=gpu:4</code>). In yambo, each GPU corresponds to a single MPI task, therefore we have still the total number of tasks equal to 8. OpenMP threading is allowed - but not too much, otherwise we lose efficiency. Here in fact, we are not using OpenMP so </code>export OMP_NUM_THREADS=1</code>.

	~~# module purge~~		In addition, you will see that in order to run on GPUs we are now using a different executable than before, obtained with a GPU-specific compilation of the code.
	~~# module load hpc~~-~~sdk/2022--binary spectrum_mpi/10.4~~.~~0--binary~~
	~~# export PATH=/m100_work/tra23_Yambo/softwares/YAMBO/5.2-gpu/bin:$PATH~~

	#In general, the gpu compilation might be different on your machine, or the executable may be the same with no need to load an additional #modules. In conclusion, we are running over 4 GPU cards, distributed with 2 MPI tasks, and using 1 thread.		module purge
			module load hpc-sdk/2022--binary spectrum_mpi/10.4.0--binary
			export PATH=/m100_work/tra23_Yambo/softwares/YAMBO/5.2-gpu/bin:$PATH

			In general, the gpu compilation might be different on your machine, or the executable may be the same with no need to load an additional modules. In conclusion, we are running over 4 GPU cards, distributed with 2 MPI tasks, and using 1 thread.

	The calculation should faster, about 2 minutes instead of 2 min and 30s with a purely CPU calculation. The gain can become even greater in larger systems. You can have a look at the results collected in folder <code>MPI2_OMP1</code>. The quasiparticle corrections are stored in human-readable form in the file <code>MPI2_OMP1.out/o-GW_bnds.qp</code>, and in netCDF format in the quasiparticle database <code>MPI2_OMP1/ndb.QP</code>.		The calculation should faster, about 2 minutes instead of 2 min and 30s with a purely CPU calculation. The gain can become even greater in larger systems. You can have a look at the results collected in folder <code>MPI2_OMP1</code>. The quasiparticle corrections are stored in human-readable form in the file <code>MPI2_OMP1.out/o-GW_bnds.qp</code>, and in netCDF format in the quasiparticle database <code>MPI2_OMP1/ndb.QP</code>.

← Older revision		Revision as of 21:45, 19 May 2025
Line 196:		Line 196:
	If you inspect it, you will see that the script adds additional variables to the yambo input file.		If you inspect it, you will see that the script adds additional variables to the yambo input file.
	These variables control the parallel execution of the code:		These variables control the parallel execution of the code:

	DIP_CPU= "1 $ncpu 1" # [PARALLEL] CPUs for each role		DIP_CPU= "1 $ncpu 1" # [PARALLEL] CPUs for each role
	DIP_ROLEs= "k c v" # [PARALLEL] CPUs roles (k,c,v)		DIP_ROLEs= "k c v" # [PARALLEL] CPUs roles (k,c,v)
	DIP_Threads= 0 # [OPENMP/X] Number of threads for dipoles		DIP_Threads= 0 # [OPENMP/X] Number of threads for dipoles
	X_and_IO_CPU= "1 1 1 $ncpu 1" # [PARALLEL] CPUs for each role		X_and_IO_CPU= "1 1 1 $ncpu 1" # [PARALLEL] CPUs for each role
	X_and_IO_ROLEs= "q g k c v" # [PARALLEL] CPUs roles (q,g,k,c,v)		X_and_IO_ROLEs= "q g k c v" # [PARALLEL] CPUs roles (q,g,k,c,v)
	X_and_IO_nCPU_LinAlg_INV= 1 # [PARALLEL] CPUs for Linear Algebra		X_and_IO_nCPU_LinAlg_INV= 1 # [PARALLEL] CPUs for Linear Algebra
	X_Threads= 0 # [OPENMP/X] Number of threads for response functions		X_Threads= 0 # [OPENMP/X] Number of threads for response functions
	SE_CPU= " 1 $ncpu 1" # [PARALLEL] CPUs for each role		SE_CPU= " 1 $ncpu 1" # [PARALLEL] CPUs for each role
	SE_ROLEs= "q qp b" # [PARALLEL] CPUs roles (q,qp,b)		SE_ROLEs= "q qp b" # [PARALLEL] CPUs roles (q,qp,b)
	SE_Threads= 0 # [OPENMP/GW] Number of threads for self-energy		SE_Threads= 0 # [OPENMP/GW] Number of threads for self-energy

	The keyword <code>DIP</code> refers to the calculations of the screening matrix elements (also called “dipoles”) needed for the screening function, <code>X</code> is the screening function (it stands for χ since it is a response function), <code>SE</code> the self-energy.		The keyword <code>DIP</code> refers to the calculations of the screening matrix elements (also called “dipoles”) needed for the screening function, <code>X</code> is the screening function (it stands for χ since it is a response function), <code>SE</code> the self-energy.
Line 218:		Line 218:
	#SBATCH --account=tra25_yambo # Charge resources used by this job to specified account		#SBATCH --account=tra25_yambo # Charge resources used by this job to specified account
	#SBATCH --time=00:10:00 # Set a limit on the total run time of the job allocation in hh:mm:ss		#SBATCH --time=00:10:00 # Set a limit on the total run time of the job allocation in hh:mm:ss
	#SBATCH --job-name=rutile # Specify a name for the job allocation		#SBATCH --job-name=rutile # Specify a name for the job allocation
	#SBATCH --partition=dcgp_usr_prod # Request a specific partition for the resource allocation		#SBATCH --partition=dcgp_usr_prod # Request a specific partition for the resource allocation
	#SBATCH --gres=tmpfs:10g # List of generic consumable resources		#SBATCH --gres=tmpfs:10g # List of generic consumable resources
	#SBATCH --qos=normal # Quality of service		#SBATCH --qos=normal # Quality of service