GW parallel strategies - Revision history

Daniele: /* Advanced: Comparing different parallelization schemes (optional) */

2023-05-30T20:46:53Z

Advanced: Comparing different parallelization schemes (optional)

← Older revision		Revision as of 20:46, 30 May 2023
Line 282:		Line 282:

	'''tips: <br>'''		'''tips: <br>'''
	* the product of the numbers entering each variable (i.e. ~~X_CPU~~ and SE_CPU) times the number of threads should always match the total number of cores (unless you want to overload the cores taking advantage of multi-threads) <br>		* the product of the numbers entering each variable (i.e. X_and_IO_CPU and SE_CPU) times the number of threads should always match the total number of cores (unless you want to overload the cores taking advantage of multi-threads) <br>
	* using the X_Threads and SE_Threads variables, you can think about setting different Hybrid schemes in between the screening and the self-energy runlevel.		* using the X_Threads and SE_Threads variables, you can think about setting different Hybrid schemes in between the screening and the self-energy runlevel.
	* memory better scales if you parallelize on bands (c v b) <br>		* memory better scales if you parallelize on bands (c v b) <br>

Daniele: /* Advanced: Comparing different parallelization schemes (optional) */

2023-05-30T20:46:28Z

Advanced: Comparing different parallelization schemes (optional)

Daniele: /* Getting familiar with yambo in parallel */

2023-05-30T20:44:55Z

Getting familiar with yambo in parallel

← Older revision		Revision as of 20:44, 30 May 2023
Line 88:		Line 88:
	DIP_ROLEs= "k c v" # [PARALLEL] CPUs roles (k,c,v)		DIP_ROLEs= "k c v" # [PARALLEL] CPUs roles (k,c,v)
	DIP_Threads= 0 # [OPENMP/X] Number of threads for dipoles		DIP_Threads= 0 # [OPENMP/X] Number of threads for dipoles
	~~X_CPU~~= "1 1 1 $ncpu 1" # [PARALLEL] CPUs for each role		X_and_IO_CPU= "1 1 1 $ncpu 1" # [PARALLEL] CPUs for each role
	~~X_ROLEs~~= "q g k c v" # [PARALLEL] CPUs roles (q,g,k,c,v)		X_and_IO_ROLEs= "q g k c v" # [PARALLEL] CPUs roles (q,g,k,c,v)
	~~X_nCPU_LinAlg_INV~~= $ncpu # [PARALLEL] CPUs for Linear Algebra		X_and_IO_nCPU_LinAlg_INV= $ncpu # [PARALLEL] CPUs for Linear Algebra
	X_Threads= 0 # [OPENMP/X] Number of threads for response functions		X_Threads= 0 # [OPENMP/X] Number of threads for response functions
	SE_CPU= " 1 $ncpu 1" # [PARALLEL] CPUs for each role		SE_CPU= " 1 $ncpu 1" # [PARALLEL] CPUs for each role

Yambowikiadmin at 09:46, 9 February 2023

2023-02-09T09:46:40Z

@@ Line 4: / Line 4: @@
 -->
 In this tutorial we will see how to setup the variables governing the parallel execution of yambo in order to perform efficient calculations in terms of both cpu time and memory to solution. As a test case we will consider the hBN 2D material. Because of its reduced dimensionality, GW calculations turns out to be very delicate. Beside the usual convergence studies with respect to k-points and sums-over-bands, in low dimensional systems a sensible amount of vacuum is required in order to treat the system as isolated, translating into a large number of plane-waves. As for other tutorials, it is important to stress that this tutorial it is meant to illustrate the functionality of the key variables and to run in reasonable time, so it has not the purpose to reach the desired accuracy to reproduce experimental results. Moreover please also note that scaling performance illustrated below may be significantly dependent on the underlying parallel architecture. Nevertheless, general considerations are tentatively drawn in discussing the results.
 == Getting familiar with yambo in parallel ==

Aferretti: /* MPI vs OpenMP scaling */

2020-01-28T16:17:37Z

MPI vs OpenMP scaling

← Older revision		Revision as of 16:17, 28 January 2020
Line 226:		Line 226:
	is printed in the report file, and can be extracted typing:		is printed in the report file, and can be extracted typing:
	$ grep "Max memory used" <report_file>		$ grep "Max memory used" <report_file>

	<!--		<!--
	If you compare for example the two extreme cases (you can use)		If you compare for example the two extreme cases (you can use)
Line 251:		Line 250:
	These last two numbers have to be compared with 0.321 (0.306)		These last two numbers have to be compared with 0.321 (0.306)
	-->		-->

	In general yambo can distribute memory when using MPI parallelism (though the actual amount depends on the distribution of MPI tasks across MPI levels).		In general yambo can distribute memory when using MPI parallelism (though the actual amount depends on the distribution of MPI tasks across MPI levels).
	Nevertheless, some memory replication is still present. In general, within the node OpenMP helps in easing the memory usage. Therefore, in cases where the node memori is		Nevertheless, some memory replication is still present. In general, within the node OpenMP helps in easing the memory usage. Therefore, in cases where the node memori is

Aferretti: /* MPI vs OpenMP scaling */

2020-01-28T16:17:05Z

MPI vs OpenMP scaling

← Older revision		Revision as of 16:17, 28 January 2020
Line 222:		Line 222:

	However, CPU time is not the only parameter you need to check.		However, CPU time is not the only parameter you need to check.
	The total memory usage is also very critical since the GW method may have a large memory footprint.		The <b>total memory usage</b> is also very critical since the GW method may have a large memory footprint.
	If you have compiled yambo with the flag <code>--enable-memory-profile</code>, the memory usage is tracked and the maximum allocated mem		If you have compiled yambo with the flag <code>--enable-memory-profile</code>, the memory usage is tracked and the maximum allocated mem
	is printed in the report file, and can be extracted typing:		is printed in the report file, and can be extracted typing:
Line 251:		Line 251:
	These last two numbers have to be compared with 0.321 (0.306)		These last two numbers have to be compared with 0.321 (0.306)
	-->		-->
	~~As you~~ can ~~see yambo is distributing~~ memory ~~since~~ the ~~single~~ MPI ~~task uses less~~ memory ~~than~~
	the ~~total one needed (you can even compare with~~ the ~~serial case)~~. ~~However~~, it is ~~not as efficient as~~		In general yambo can distribute memory when using MPI parallelism (though the actual amount depends on the distribution of MPI tasks across MPI levels).
	OpenMP ~~in doing so~~.		Nevertheless, some memory replication is still present. In general, within the node OpenMP helps in easing the memory usage. Therefore, in cases where the node memori is
			tight, one may consider changing some MPI tasks for OpenMP threads within the node.


	Using a hybrid scheme you may also consider running yambo on mode than one node.		Using a hybrid scheme you may also consider running yambo on mode than one node.

Aferretti: /* MPI vs OpenMP scaling */

2020-01-28T15:59:45Z

MPI vs OpenMP scaling

← Older revision		Revision as of 15:59, 28 January 2020
Line 227:		Line 227:
	$ grep "Max memory used" <report_file>		$ grep "Max memory used" <report_file>

	<--!		<!--
	If you compare for example the two extreme cases (you can use)		If you compare for example the two extreme cases (you can use)
	$ grep Gb run_MPI1_OMP/l \| grep Alloc (use this for the case with only one MPI proc)		$ grep Gb run_MPI1_OMP/l \| grep Alloc (use this for the case with only one MPI proc)
Line 250:		Line 250:
	Multiplying by 16 you obtain an estimate of the total memory: 0.09116=1.456 (0.07616=1.216)		Multiplying by 16 you obtain an estimate of the total memory: 0.09116=1.456 (0.07616=1.216)
	These last two numbers have to be compared with 0.321 (0.306)		These last two numbers have to be compared with 0.321 (0.306)
	-->		-->
	As you can see yambo is distributing memory since the single MPI task uses less memory than		As you can see yambo is distributing memory since the single MPI task uses less memory than
	the total one needed (you can even compare with the serial case). However, it is not as efficient as		the total one needed (you can even compare with the serial case). However, it is not as efficient as

Aferretti: /* MPI vs OpenMP scaling */

2020-01-28T15:58:40Z

MPI vs OpenMP scaling

← Older revision		Revision as of 15:58, 28 January 2020
Line 222:		Line 222:

	However, CPU time is not the only parameter you need to check.		However, CPU time is not the only parameter you need to check.
	~~Also~~, the ~~total~~ memory usage is ~~important~~		The total memory usage is also very critical since the GW method may have a large memory footprint.
			If you have compiled yambo with the flag <code>--enable-memory-profile</code>, the memory usage is tracked and the maximum allocated mem
			is printed in the report file, and can be extracted typing:
			$ grep "Max memory used" <report_file>

			<--!
	If you compare for example the two extreme cases (you can use)		If you compare for example the two extreme cases (you can use)
	$ grep Gb run_MPI1_OMP/l \| grep Alloc (use this for the case with only one MPI proc)		$ grep Gb run_MPI1_OMP/l \| grep Alloc (use this for the case with only one MPI proc)
Line 245:		Line 250:
	Multiplying by 16 you obtain an estimate of the total memory: 0.09116=1.456 (0.07616=1.216)		Multiplying by 16 you obtain an estimate of the total memory: 0.09116=1.456 (0.07616=1.216)
	These last two numbers have to be compared with 0.321 (0.306)		These last two numbers have to be compared with 0.321 (0.306)
			-->
	As you can see yambo is distributing memory since the single MPI task uses less memory than		As you can see yambo is distributing memory since the single MPI task uses less memory than
	the total one needed (you can even compare with the serial case). However, it is not as efficient as		the total one needed (you can even compare with the serial case). However, it is not as efficient as
Line 259:		Line 265:

	'''tips:''' <br>		'''tips:''' <br>
	- in real life calculations running on n_cores > 100, it is a good idea to adopt a hybrid approach <br>		* in real life calculations running on n_cores > 100, it is a good idea to adopt a hybrid approach <br>
	- with OpenMP, you cannot exit the single node, with MPI you can		* with OpenMP, you cannot exit the single node, with MPI you can

	== Advanced: Comparing different parallelization schemes (optional) ==		== Advanced: Comparing different parallelization schemes (optional) ==

Aferretti: /* MPI vs OpenMP scaling */

2020-01-28T15:18:52Z

MPI vs OpenMP scaling

← Older revision		Revision as of 15:18, 28 January 2020
Line 211:		Line 211:
	Parsing the data we will obtain something similar to		Parsing the data we will obtain something similar to

	# ncores MPI threads ~~Dipoles~~ Xo X Sgm_x Sgm_c WALL_TIME		# ncores MPI threads dip Xo X io_X io_WF Sgm_x Sgm_c WALL_TIME
	16 ~~1 16 1~~.~~00s 38~~.~~00s~~ 0.~~00s 2~~.~~00s 33.00s 01m25s~~		16 2 8 3.6816s 386.0552s 0.0856s 0.0268s 0.2163s 18.9077s 530.0875s 15m43s
	~~16 2 8 2.00s 34.00s~~ 0.~~00s 1~~.~~00s 16~~.~~00s 01m06s~~		16 4 4 0.9950s 82.5070s 0.1098s 0.0299s 0.2051s 3.0837s 122.6565s 03m32s
	16 4 4 0.~~00s 34~~.~~00s~~ 0.~~00s 1~~.~~00s 11~~.~~00s 56s~~		16 8 2 0.8524s 72.6708s 0.0986s 0.0293s 0.2282s 3.0379s 88.1589s 02m48s
	16 8 2 0.~~00s 33~~.~~00s~~ 0.~~00s~~ 0.~~00s 9~~.~~00s 52s~~		16 16 1 0.7653s 68.2550s 0.1048s 0.0309s 0.2463s 2.9211s 72.6220s 02m27s
	16 16 1 0.~~00s 35~~.~~00s~~ 0.~~00s~~ 0.~~00s 7~~.~~00s 54s~~

	As you can see here the total CPU time decreases more and more moving the parallelization from the OpenMP to the MPI level.		As you can see here the total CPU time decreases more and more moving the parallelization from the OpenMP to the MPI level.
	Sigma_c in particular scales better. ~~However~~, the ~~fastest run is not~~ the ~~one with 16 MPI task but~~ the ~~8 2 configuration~~		Sigma_c in particular scales better. Nevertheless, note that the relative performance of the different parallel configurations
	~~since Xo is faster using a hybrid MPI-OpenMP scheme~~.		may strongly depend on the actual machine you are running on.

	However, CPU time is not the only parameter you need to check.		However, CPU time is not the only parameter you need to check.

Aferretti: /* Pure OpenMP scaling */

2020-01-28T14:44:36Z

Pure OpenMP scaling

← Older revision		Revision as of 14:44, 28 January 2020
Line 180:		Line 180:
	Since we are already using 16 threads, we cannot also distribute among MPI tasks, i.e. ncpu will result equal to 1.		Since we are already using 16 threads, we cannot also distribute among MPI tasks, i.e. ncpu will result equal to 1.
	Try setting nthreads to 16, 8, 4 and 2 and again to plot the execution time vs the number of threads using the python script.		Try setting nthreads to 16, 8, 4 and 2 and again to plot the execution time vs the number of threads using the python script.
	Again you should be able to produce data similar to the following		Again you should be able to produce data similar to the following. In the following we stopped increasing the number of threads up to 8 because of the
			specific architecture used in the runs (a dual socket machine with 8 cores/socket).

	# ncores threads dip Xo X io_X io_WF Sgm_x Sgm_c WALL_TIME		# ncores threads dip Xo X io_X io_WF Sgm_x Sgm_c WALL_TIME

← Older revision		Revision as of 20:46, 30 May 2023
Line 256:		Line 256:
	== Advanced: Comparing different parallelization schemes (optional) ==		== Advanced: Comparing different parallelization schemes (optional) ==

	Up to now we used the default parallelization scheme.		Up to now, we used the default parallelization scheme.
	Yambo also allows you to tune the parameters which controls the parallelization scheme.		Yambo also allows you to tune the parameters which controls the parallelization scheme.
	To this end you can open again the job.sh script and modify the section where the yambo input		To this end, you can open again the job.sh script and modify the section where the yambo input
	variables are set		variables are set

	~~X_CPU~~= "<span style="color:red">1 1 1 $ncpu 1</span>" # [PARALLEL] CPUs for each role		X_and_IO_CPU= "<span style="color:red">1 1 1 $ncpu 1</span>" # [PARALLEL] CPUs for each role
	~~X_ROLEs~~= "q g k c v" # [PARALLEL] CPUs roles (q,g,k,c,v)		X_and_IO_ROLEs= "q g k c v" # [PARALLEL] CPUs roles (q,g,k,c,v)
	#~~X_nCPU_LinAlg_INV~~= $ncpu # [PARALLEL] CPUs for Linear Algebra		#X_and_IO_nCPU_LinAlg_INV= $ncpu # [PARALLEL] CPUs for Linear Algebra
	X_Threads= 0 # [OPENMP/X] Number of threads for response functions		X_Threads= 0 # [OPENMP/X] Number of threads for response functions
	DIP_Threads= 0 # [OPENMP/X] Number of threads for dipoles		DIP_Threads= 0 # [OPENMP/X] Number of threads for dipoles
Line 270:		Line 270:
	SE_Threads= 0		SE_Threads= 0

	In particular "~~X_CPU~~" sets how the MPI Tasks are distributed in the calculation of the response function.		In particular, "X_and_IO_CPU" sets how the MPI Tasks are distributed in the calculation of the response function.
	The possibilities are shown in the "~~X_ROLEs~~". The same holds for "SE_CPU" and "SE_ROLEs" which control		The possibilities are shown in the "X_and_IO_ROLEs". The same holds for "SE_CPU" and "SE_ROLEs" which control
	how MPI Tasks are distributed in the calculation of the response function.		how MPI Tasks are distributed in the calculation of the response function.

	Please try different parallelization schemes and check the performances of Yambo.		Please try different parallelization schemes and check the performances of Yambo.
	In doing so you should also change the jobname in the run.sh script		In doing so, you should also change the jobname in the run.sh script
	label=MPI${ncpu}_OMP${nthreads}<span style="color:red">_scheme1</span>		label=MPI${ncpu}_OMP${nthreads}<span style="color:red">_scheme1</span>

	Using the python script, you can then ~~chenck~~ how speed, memory and load balance between the CPUs are affected.		Using the python script, you can then check how speed, memory and load balance between the CPUs are affected.
	For more details see also the [[Using_Yambo_in_parallel\|Parallel module]]		For more details, see also the [[Using_Yambo_in_parallel\|Parallel module]]

	'''tips: <br>'''		'''tips: <br>'''
	* the product of the numbers entering each variable (i.e. X_CPU and SE_CPU) times the number of threads should always match the total number of cores (unless you want to overload the cores taking advantage of multi-threads) <br>		* the product of the numbers entering each variable (i.e. X_CPU and SE_CPU) times the number of threads should always match the total number of cores (unless you want to overload the cores taking advantage of multi-threads) <br>
	* using the X_Threads and SE_Threads variables you can think about setting different Hybrid schemes in between the screening and the self-energy runlevel.		* using the X_Threads and SE_Threads variables, you can think about setting different Hybrid schemes in between the screening and the self-energy runlevel.
	* memory better scales if you parallelize on bands (c v b) <br>		* memory better scales if you parallelize on bands (c v b) <br>
	* parallelization on k-points performs similarly to parallelization on bands, but memory requires more memory <br>		* parallelization on k-points performs similarly to parallelization on bands, but memory requires more memory <br>
	* parallelization on q-points requires much less communication in between the MPI tasks. It ~~maybe~~ useful if you run on more than one node and the ~~inter-node~~ connection is slow		* parallelization on q-points requires much less communication in between the MPI tasks. It may be useful if you run on more than one node and the internode connection is slow

	<br>		<br>