Recent Posts

1

Escf and Egrad / Re: Egrad

« Last post by Pingping on Today at 04:54:41 PM »

hi!
I recently encountered the same error message:

 Warning! No convergence within  35 iterations.
 Unless you have specified a very low $escfiterlimit/$forceiterlimit,
 this is a reason to worry!



 IRREP   Vector     Eigenvalue           Euclidean residual norm

  a         1    1.174219598903526D-01    2.005914809903496D-04


 cannot deallocate irrep_tensor_pack object that was never fully allocated!

========================
 internal module stack:
------------------------
    egrad
========================

  abnormal termination
 egrad ended abnormally
error in gradient step (1)
I put the attachment underneath,Look forward to your reply. Thank you!

2

Miscellaneous / Re: Heavy elements and relativistic effects

« Last post by JakubV on March 18, 2023, 10:07:41 PM »

P.S. II: So I have tried the ROHF + soghf and faced this error.
How can I work around this, please?

Code: [Select]

              RI-J - INFORMATION
           ------------------------
 Contributions to RI integral batches:
  neglected integral batches:                     0
  direct contribution:                            0
  memory contribution:                   561
 Memory core needed for (P|Q) and Cholesky      1 MByte
 Memory core minimum needed except of (P|Q)     1 MByte
 Total minimum memory core needed (sum)         1 MByte

 ****************************************
 Memory allocated for RI-J    34 MByte
 ****************************************


 Difference densities switched on

 Exact HF-exchange is used

 Starting SCF iterations


 ITERATION  ENERGY          1e-ENERGY        2e-ENERGY     NORM[dD(SAO)]  TOL
     1  -484.39648956847    -782.13430779     289.39910750    0.000D+00 0.203D-11
                            Ex   [ex-K]  = -49.5291743869
                            Coul [RI-J]   =  338.928281885
                            E[SO]        =  0.00000000000
Subroutine ddmat can not establish optimal linear combination of delta densities !
 Next fock matrix will be calculated from scratch !
Subroutine ddmat can not establish optimal linear combination of delta densities !
 Next fock matrix will be calculated from scratch !
                                              current damping :  2.000

 ITERATION  ENERGY          1e-ENERGY        2e-ENERGY     NORM[dD(SAO)]  TOL
     2                   NaN              NaN              NaN0.000D+00 0.203D-11
                            Ex   [ex-K]  =                NaN
                            Coul [RI-J]   =                NaN
                            E[SO]        =                NaN
Subroutine ddmat can not establish optimal linear combination of delta densities !
 Next fock matrix will be calculated from scratch !
Subroutine ddmat can not establish optimal linear combination of delta densities !
 Next fock matrix will be calculated from scratch !
                                              current damping :  2.050

 ITERATION  ENERGY          1e-ENERGY        2e-ENERGY     NORM[dD(SAO)]  TOL
     3                   NaN              NaN              NaN0.000D+00 0.203D-11
                            Ex   [ex-K]  =                NaN
                            Coul [RI-J]   =                NaN
                            E[SO]        =                NaN
Subroutine ddmat can not establish optimal linear combination of delta densities !
 Next fock matrix will be calculated from scratch !
Subroutine ddmat can not establish optimal linear combination of delta densities !
 Next fock matrix will be calculated from scratch !
                                              current damping :  2.100
... and so on, there are only NaNs onward.

control file

Code: [Select]

$title
$symmetry c1
$user-defined bonds    file=coord
$coord    file=coord
$optimize
 internal   off
 redundant  off
 cartesian  on
 global     off
 basis      off
$atoms
u  1                                                                           \
   basis =u Cao-2c                                                             \
   ecp   =u def-ecp-2c                                                         \
   jbas  =u def-TZVPP
h  2                                                                           \
   basis =h aug-cc-pVTZ-2c                                                     \
   jbas  =h universal
$basis    file=basis
$ecp    file=basis
$scfmo   file=mos
$closed shells
 a       1-14                                   ( 2 )
$scfiterlimit     8000
$scfconv        8
$thize     0.10000000E-04
$thime        5
$scfdamp   start=2.000  step=0.050  min=0.50
$scfdump
$scfintunit
 unit=30       size=0        file=twoint
$scfdiis
$maxcor    500 MiB  per_core
$scforbitalshift  closedshell=.4
$drvopt
   cartesian  on
   basis      off
   global     off
   hessian    on
   dipole     on
   nuclear polarizability
$interconversion  off
   qconv=1.d-7
   maxiter=25
$coordinateupdate
   dqmax=0.3
   interpolate  on
   statistics    5
$forceupdate
   ahlrichs numgeo=0  mingeo=3 maxgeo=4 modus=<g|dq> dynamic fail=0.3
   threig=0.005  reseig=0.005  thrbig=3.0  scale=1.00  damping=0.0
$forceinit on
   diag=default
$energy    file=energy
$grad    file=gradient
$forceapprox    file=forceapprox
$soghf
$ricore      800
$rij
$jbas    file=auxbasis
$rundimensions
   natoms=2
   nbf(CAO)=164
   nbf(AO)=127
$last step     ridft
$charge from ridft
            NaN (not to be modified here)
$dipole from ridft
  x                  NaN    y                  NaN    z                  NaN    a.u.
   | dipole | =             NaN  debye
$spinor_real       file=spinor.r
$spinor_imag       file=spinor.i
$spinor shells
 a       1-14                                   ( 2 )
 a       19-20                                  ( 1 )
$end

coord

Code: [Select]

$coord
    0.00000000000000      0.00000000000000     -1.91876184798602      u
    0.00000000000000      0.00000000000000      1.91876184798602      h
$user-defined bonds
$end


3

Miscellaneous / Relativistic effects, Kramers unrestricted

« Last post by JakubV on March 18, 2023, 02:27:40 AM »

Dear Turbomole Experts,

I would like to investigate molecules like UH+ (or neutral UH), which have several open shells (4 or 5 in scalar treatment), also with spin-orbit splitting included and possibly at the DFT level.

Combining UHF and spin-orbit (soghf) - is it somehow possible, please? As that would mean - Kramers unrestricted SCF/DFT...
Is there any Kramers unrestricted method in Turbomole, or could you hint to me software capable of that, please?

ROHF and soghf shall be possible to combine, but (possibly also since DFT is not well defined for ROHF) TM will not allow me to combine ROHF and DFT anyway.

Best regards,
JakubV

P.S.:  I apologise if it is somehow clear from the documentation.

4

Parallel Runs / Re: TURBOMOLE + Slurm 22.05: MPI not working?

« Last post by uwe on March 07, 2023, 11:20:21 PM »

Hello,

there is no 'official' way when it comes to queuing systems. We also use SLURM and do not have similar issues, but we just use the default settings.

It looks like the Intel MPI installation that is shipped with TURBOMOLE (in mpirun_scripts/IMPI) is somehow (at least in our setup) borked.

Intel MPI in Turbomole is not changed or modified in any way, it is simply 'pure'. So I do not see any reason why it should behave differently in Turbomole.
If you have another application on your cluster that runs fine with Intel MPI, using the very same settings might help.

But as we are using cgroups to limit access to CPUs these srun-options have the effect that if the job is configured to run more than one task per node, all those tasks of one node run on the very same single CPU!

There have indeed been several cases where cgroups and cpuset (and most likely any other tool that tries to pin tasks to CPU cores) caused similar problems when running MPI jobs. Intel MPI by default enables process pinning. So if SLURM, cgroups and Intel MPI all try to pin the processes to CPU cores in an unlucky way, it might well happen that they all run on the same core... We use the default Intel MPI pinning and do not have anything else in use, perhaps this helps to avoid such problems. But it seems that not using cgroups is not an option for you.

Did you try to set I_MPI_PIN=0 to disable pinning?

5

Parallel Runs / Re: TURBOMOLE + Slurm 22.05: MPI not working?

« Last post by hermannschwaerzler on March 07, 2023, 04:37:11 PM »

Hi uwe,

thanks for your confirmation that I went in the right direction. :-)
Loading that module in mpirun does imo essentially replace the IntelMPI installation of TURBOMOLE as it prepends PATH, LD_LIBRARY_PATH and others with directories that point to the working IntelMPI version.

So this is the official way of doing things?
Why does the provided version not work out of the box?

In our case I had to go one step further and had to add

export TM_MPIADDOPT="-bootstrap ssh"

to my job-script in order to make IntelMPI use all the CPUs of the node it was running on (without this option it was running all processes on one single CPU). Unfortunately this works only when one uses only one node but my "customers" are happy with that.
When one uses more than one node with this option the processes on further nodes are placed there outside the control of Slurm (with ssh) which is a no-go for us.

Regards,
Hermann

6

Parallel Runs / Re: TURBOMOLE + Slurm 22.05: MPI not working?

« Last post by uwe on March 05, 2023, 12:36:22 PM »

Hi,

did you try to completely replace the Intel MPI version that comes with Turbomole by your local one? I tried the MPI binaries with both Intel MPI version 2019 and 2021 and the runtime libraries seem to be compatible. Just remove the $TURBODIR/mpirun_scripts/IMPI content (should be just a intel64 folder) and link intel64 to your local Intel MPI version.

Turbomole is not using anything 'special' when it comes to MPI, so if your Intel MPI works well with your queuing system, it should also run Turbomole correctly.

7

Parallel Runs / Re: TURBOMOLE + Slurm 22.05: MPI not working?

« Last post by hermannschwaerzler on March 03, 2023, 04:32:21 PM »

OK, I found the problem and kind of a solution:

It looks like the Intel MPI installation that is shipped with TURBOMOLE (in mpirun_scripts/IMPI) is somehow (at least in our setup) borked.
As we have our own version of Intel MPI available on our systems I "fixed" this by "patching" mpirun_scripts/IMPI/intel64/bin/mpirun such, that the corresponding module of our system is loaded right before mpiexec.hydra is called (around line 66).
The hydra_pmi_proxy that is used now is doing the right thing and starts the appropriate number of (in my case) ridft_mpi processes.

So it does work now, but with one major limitation: mpiexec.hydra is running in a way that it uses

srun -N 1 -n 1 ...

to start a hydra_bstrap_proxy that in turn starts said hydra_pmi_proxy which starts the processes that do the calculation. But as we are using cgroups to limit access to CPUs these srun-options have the effect that if the job is configured to run more than one task per node, all those tasks of one node run on the very same single CPU!
Which is obviously a performance problem.

So I guess I am still missing something?

8

Parallel Runs / Re: TURBOMOLE + Slurm 22.05: MPI not working?

« Last post by hermannschwaerzler on March 03, 2023, 10:06:55 AM »

Thanks for the hint.
I tried that. There are some changes visible but the basic problem still exists:

Code: [Select]

`- slurmstepd: [63509.batch]                                                                                                                                                                                                                                                                                           
     `- /bin/bash /var/spool/slurm/slurmd/job63509/slurm_script                                                                                                                                                                                                                                                         
         `- /bin/bash /path/to/TURBOMOLE/bin/em64t-unknown-linux-gnu_mpi/ridft                                                                                                                                                                                                                                 
             `- /bin/sh /path/to/TURBOMOLE/mpirun_scripts/IMPI/intel64/bin/mpirun -genv OMP_NUM_THREADS=1 -genv TURBODIR=/path/to/TURBOMOLE -genv I_MPI_PIN=off -genv OMP_STACK_SIZE=256M -genv LD_LIBRARY_PATH=/path/to/TURBOMOLE/mpirun_scripts/IMPI/intel64/libfabric/lib:/gpfs/g+
                 `- mpiexec.hydra -genv OMP_NUM_THREADS=1 -genv TURBODIR=/path/to/TURBOMOLE -genv I_MPI_PIN=off -genv OMP_STACK_SIZE=256M -genv LD_LIBRARY_PATH=/path/to/TURBOMOLE/mpirun_scripts/IMPI/intel64/libfabric/lib:/path/to/TURBOMOLE/mpirun_scripts/IMPI/intel+
                     `- /usr/slurm/bin/srun -N 1 -n 1 --nodelist n054 --input none /path/to/TURBOMOLE/mpirun_scripts/IMPI/intel64/bin//hydra_bstrap_proxy --upstream-host n054.domain --upstream-port 34219 --pgid 0 --launcher slurm --launcher-number 1 --base-path /path/to/TURBOM+
                         `- /usr/slurm/bin/srun -N 1 -n 1 --nodelist n054 --input none /path/to/TURBOMOLE/mpirun_scripts/IMPI/intel64/bin//hydra_bstrap_proxy --upstream-host n054.domain --upstream-port 34219 --pgid 0 --launcher slurm --launcher-number 1 --base-path /path/to/TU+
 `- slurmstepd: [63509.0]                                                                                                                                                                                                                                                                                               
     `- /path/to/TURBOMOLE/mpirun_scripts/IMPI/intel64/bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9                                                                                                                                                                                       

We have a cluster (not the central one) in our Chemistry department where this works. If it works the hydra_pmi_proxy starts the requested number of ridft_mpi processes. In our case as you see no such process is started by hydra_pmi_proxy.

So sorry no, MPI_USESRUN=1 did not help. :-(

9

Parallel Runs / Re: TURBOMOLE + Slurm 22.05: MPI not working?

« Last post by uwe on March 02, 2023, 09:28:28 PM »

Hello,

please set:

export MPI_USESRUN=yes

in your submit script before calling ridft and check if that helps.

Why this might help:

What Turbomole does in case of parallel MPI jobs is to call a script instead of the binary, so 'ridft' is here just a script that checks for the queuing system and sets the environment (see $TURBODIR/bin/em64t-unknown-linux-gnu_mpi/ridft).  It is actually the same script for all Turbomole modules that run in parallel using MPI.

In there you will find the section that detects SLURM, the lines start with:

#
# SLURM environment
#
if [ $DONE -eq 0 -a -n "${SLURMD_NODENAME}" ]; then
   # SLURM detected


If srun is configured by the queuing system, Intel MPI uses it automatically, but the Turbomole script creates by default a file with the name of all nodes. This is named NodeFile.<PID_of_the_script>, in your case NodeFile.1604633 and passed to the mpirun command using the command line option -machinefile.

In your output that's how the lines:
RUNNING PROGRAM /path/to/TURBOMOLE/bin/em64t-unknown-linux-gnu_mpi/ridft_mpi.
/path/to/TURBOMOLE/mpirun_scripts/IMPI/intel64/bin/mpirun -machinefile NodeFile.1604633 -genv OMP_NUM_THREADS=1 -genv TURBODIR=/path/to/TURBOMOLE/[...]
are generated.

In case of srun the -machinefile option might be not helpful and prevents Intel MPI to start the right number of process on the right nodes.

Setting $MPI_USESRUN to some value (it just should not be empty) avoids adding the -machinefile option.

10

Parallel Runs / TURBOMOLE + Slurm 22.05: MPI not working?

« Last post by hermannschwaerzler on February 28, 2023, 03:57:26 PM »

Hi everybody,

we are having a problem wrt running MPI parallelisation on our cluster (Slurm 22.05 - TURBOMOLE 7.5 and 7.7 Demo).
As data we are using the example from the tutorial (benzene) and as a job-script this:

Code: [Select]

#!/bin/bash
#SBATCH --job-name=turbomole-test
#SBATCH --nodes=1
#SBATCH --ntasks=4
#SBATCH --cpus-per-task=1
#SBATCH --hint=nomultithread
#SBATCH --mem-per-cpu=10G
#SBATCH --time=00:30:00

export TURBODIR=/path/to/TURBOMOLE
export PATH=$TURBODIR/scripts:$TURBODIR/mpirun_scripts:$PATH

## set locale to C
unset LANG
unset LC_CTYPE

# set stack size limit to unlimited:
ulimit -s unlimited

# Set environment variables for an MPI job
export PARA_ARCH=MPI
export PATH="${TURBODIR}/bin/`sysname`:${PATH}"
export PARNODES=$SLURM_NTASKS

ridft

When we submit this job it seems to run, but the only output that is produced is this:

Code: [Select]

STARTING ridft VIA YOUR QUEUING SYSTEM!
RUNNING PROGRAM /path/to/TURBOMOLE/bin/em64t-unknown-linux-gnu_mpi/ridft_mpi.
/path/to/TURBOMOLE/mpirun_scripts/IMPI/intel64/bin/mpirun -machinefile NodeFile.1604633 -genv OMP_NUM_THREADS=1 -genv TURBODIR=/path/to/TURBOMOLE -genv I_MPI_PIN=off -genv OMP_STACK_SIZE=256M -genv LD_LIBRARY_PATH=/path/to/TURBOMOLE/mpirun_scripts/IMPI/intel64/libfabric/lib:/path/to/TURBOMOLE/mpirun_scripts/IMPI/intel64/lib/release:/path/to/TURBOMOLE/mpirun_scripts/IMPI/intel64/lib:/path/to/TURBOMOLE/libso/em64t-unknown-linux-gnu_mpi /path/to/TURBOMOLE/bin/em64t-unknown-linux-gnu_mpi/ridft_mpi

When we check with top on the node where it is running we see the situation in the attachment:

• ridft has detected that it has to use srun (which is good!).
• But it uses "-N 1 -n 1" i.e. it requests one node and only one task (although we asked for 4 in our job-script).
• There is a hydra_pmi_proxy running that uses 100% of one CPU (but produces no output).

Why is this not working?
Are we missing anything?