Actuially, simple dscf fails after the statistics run and I am getting the same errors. The trick here that ulimit -a is fine on the computational nodes so not sure what to do:
[jbaltrus@compute-4-111:~]$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 204800
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) unlimited
cpu time (seconds, -t) unlimited
max user processes (-u) 1000
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Warning: no access to tty (Bad file descriptor).
Thus no job control in this shell.
which: no prsh in (/opt/turbomole62/TURBOMOLE/bin/em64t-unknown-linux-gnu_mpi:/opt/turbomole62/TURBOMOLE/scripts:/opt/gridengine/bin/lx26-amd64:/usr/java/latest/bin:/opt/fips/bin:/tmp/16648.1.UI:/opt/turbomole62/TURBOMOLE/bin/em64t-unknown-linux-gnu:/opt/turbomole62/TURBOMOLE/scripts:/opt/gridengine/bin/lx26-amd64:/usr/kerberos/bin:/usr/java/latest/bin:/opt/intel/cce/10.0/bin:/opt/fsl/bin:/opt/fips:/opt/brains2/bin:/usr/local/bin:/bin:/usr/bin:/opt/freesurfer/bin:/opt/freesurfer/bin/Linux:/opt/freesurfer/fsfast/bin:/opt/freesurfer/fsfast/bin/Linux:/opt/freesurfer/afni/Linux:/opt/freesurfer/bin/noarch:/opt/freesurfer/local/bin/Linux:/opt/ganglia/bin:/opt/ganglia/sbin:/opt/kent:/opt/ncbi/build:/opt/rocks/bin:/opt/rocks/sbin:/Users/jbaltrus/bin:./:/opt/crystal09/bin/Linux-ifort_11.1_openmpi_emt64/v1_0_1:/Users/jbaltrus:/opt/crystal09/crgra2006:/usr/X11R6/bin:/opt/bio/ncbi/bin:/opt/bio/mpiblast/bin/:/opt/bio/EMBOSS/bin:/opt/bio/clustalw/bin:/opt/bio/t_coffee/bin:/opt/bio/phylip/exe:/opt/bio/mrbayes:/opt/bio/fasta:/opt/bio/glimmer/bin:/opt/bio/glimmer/scripts:/opt/bio/gmap/bin:/opt/bio/gromacs/bin:/opt/bio/autodocksuite/bin:/opt/freesurfer/bin:/opt/freesurfer/bin/Linux:/opt/freesurfer/fsfast/bin:/opt/freesurfer/fsfast/bin/Linux:/opt/freesurfer/afni/Linux:/opt/freesurfer/bin/noarch:/opt/freesurfer/local/bin/Linux:/opt/ganglia/bin:/opt/ganglia/sbin:/opt/rocks/bin:/opt/rocks/sbin)
dscf ended normally
libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
This will severely limit memory registrations.
libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
This will severely limit memory registrations.
libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
This will severely limit memory registrations.
libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
This will severely limit memory registrations.
libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
This will severely limit memory registrations.
libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
This will severely limit memory registrations.
libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
This will severely limit memory registrations.
libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
This will severely limit memory registrations.
libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
This will severely limit memory registrations.
libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
This will severely limit memory registrations.
libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
This will severely limit memory registrations.
libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
This will severely limit memory registrations.
libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
This will severely limit memory registrations.
libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
This will severely limit memory registrations.
libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
This will severely limit memory registrations.
libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
This will severely limit memory registrations.
libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
This will severely limit memory registrations.
libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
This will severely limit memory registrations.
dscf_mpi: Rank 0:2: MPI_Init: ibv_create_cq() failed
dscf_mpi: Rank 0:2: MPI_Init: Can't initialize RDMA device
dscf_mpi: Rank 0:2: MPI_Init: MPI BUG: Cannot initialize RDMA protocol
MPI Application rank 2 exited before MPI_Init() with status 1
forrtl: error (78): process killed (SIGTERM)
forrtl: error (78): process killed (SIGTERM)
forrtl: error (78): process killed (SIGTERM)
forrtl: error (78): process killed (SIGTERM)
forrtl: error (78): process killed (SIGTERM)
forrtl: error (78): process killed (SIGTERM)
forrtl: error (78): process killed (SIGTERM)
forrtl: error (78): process killed (SIGTERM)
forrtl: error (78): process killed (SIGTERM)
forrtl: error (78): process killed (SIGTERM)
forrtl: error (78): process killed (SIGTERM)
forrtl: error (78): process killed (SIGTERM)
forrtl: error (78): process killed (SIGTERM)
forrtl: error (78): process killed (SIGTERM)
forrtl: error (78): process killed (SIGTERM)
forrtl: error (78): process killed (SIGTERM)