RICC2/RIMP2 for extra large systems

[kabel999]

Hello, we would like calculate the RI-MP2 Single-Point energies for systems containing ~400 atoms  (~5000 basis functions) on 4-processor Xeon machine with 32GB of RAM with Turbomole 6.0
Starting the single-processor job with less than 16 GB of memory works fine using rimp2 as well as ricc2 module (with option mp2energy only). In the manual is written than the ricc2 algorithm is more efficient than the rimp2 one in dealing with large systems how big of CPU saving using ricc2 algorithm can be expected?   What is the difference between these algorithms?
However, we met several problems during the calculations:
Firstly, trying to specify more than 16GB of memory both RIMP2 and RICC2 modules crashed (even for much smaller systems....)  with the same message:
 " Allocation failure for irreducible tensor analysis !"
Secondly , trying to run parallel ricc2 calculation (while for small system it was working correctly)
the job crashed with the folllowing mesage:

  Number of symmetry-nonredundant auxiliary basis functions:    14016

     Block lengths for integral files:
        frozen occupied (BOI):       26 Mbyte
        active occupied (BJI):       70 Mbyte
        active virtual  (BAI):      300 Mbyte
        frozen virtual  (BGI):        0 Mbyte
               general  (BTI):      395 Mbyte

   =========================================================================



 ========================
  internal module stack:
 ------------------------
    ricc2
    ccmkbaoi
 ========================

 Unknown error in cc_bmpiresort
 ricc2 ended abnormally
 ricc2 ended abnormally
MPI Application rank 0 exited before MPI_Finalize() with status 13
forrtl: error (78): process killed (SIGTERM)

What would be more efficient - to run the job on 4 proccessors (each with 8 GB memory) , or 2x16Gb or 1x32GB?
And the last quenstion: can be the maximum size of scratch disk space specified during the ricc2 procedure? I found in the manual how to handle it only in scf and mp2 jobs....

Thank you very much for your help!
Martin

[christof.haettig]

Hello,

1.) Both rimp2 and ricc2 use presently still standard integer (integer*4) variables in the calculations of memory demands etc.
Therefore both programs crash due to an integer overflow if you specify more 16 Gb or more in $maxcor.

2.) The performance advantage of ricc2 compared to rimp2 depends on the system size and the symmetry. ricc2 exploits
      some additional symmetry for abelian point groups and uses a different multipass algorithm for the integral evaluation
      and the N^5 scaling steps.

3.) The failure in cc_bmpiresort is caused by a limitation for a buffer size used in the communication. You can increase it by
      setting in the control file
                 $mpi_param
                       mpi_maxlen 134217728
     mpi_maxlen has to be given in bytes. default are 64 Mb, the above setting corresponds to 128 Mb. cc_bmpiresort requires
     that mpi_maxlen is at least 8*nrbfx*nsbfmax, where nrbfx is the number of auxiliary basis functions, nsbfmax the max. number
     of AOs assigned to a given slave process (approx. total number of AOs divided by the number of processors)

4.) 1 processor with 32 Gb does presently not work. if running with 2 processors with 16 Gb means that you leave to processors idle,
      4 processors with 8 Gb will be more efficient.

5.) for RI-MP2 calculations with ricc2 (or rimp2) there is presently no option to reduce the disc space requirements. But on computer
      clusters (not a single multiprocessor machine) one can exploit that in parallel calculations (with more than 3 processes) the
     individual slave processes will need less disc space than a sequential calculation.

Christof

                     

[msulka]

Hello,

regarding the point 5.), I was wondering whether there is still no option to limit disk space requirements within ricc2 calculations.
I've got limitations on disk 200 GB and cannot therefore run big ricc2(mp2) calculations.
Thank you for any help.

Martin

[Hauke]

There is a line called "disk space limit (MB)" in the output of ricc2 (TM 6.3.1) but don't know how to set it to values that are different from "none".

So I can't help you with limiting the disk space requirements. But do you know about the possibility to use the $tmpdir /scratch/thisjob in the controlfile to place the big intermediate files in a different location? On a computer cluster there is usually some directory like /scratch /scr  or whatever it is called. Maybe you can find out where such scratch file system is located and then place the files there. These days 200 GB scratch is not very much.

[msulka]

Thanks for the reply.
I was trying to modify that line "disk space limit (MB)" but without any success, it still remains "none". Probably this option is not implemented yet.
200GB scratch is, of course very small, but that's how it is on the cluster I'm using... So the solution for me was to ask for the extra scratch (now I've got 2 TB).

[christof.haettig]

There is no way to restricted the disc space for ricc2. The reason is that ricc2 anyway only dumps MO integrals to disc. It doesn't use the disc as buffer for AO integrals for which it could switch between direct and non-direct mode.

The only way you can reduce the disc space demands per machine is to run calculations MPI parallel and distribute them over several machines.

Christof