CADESUser DocumentationSHPC Condo User GuideRunning JobsSlurm Resource Queues


NOTICE: Nodes belonging to the birthright, chem, cnms, ccsd, bsd, mstd, ntrc, nucphys, virtues, theory, qmc, and ccsi groups have now moved to the Slurm scheduler and can be accessed from or-slurm-login.ornl.gov. See the Running Jobs documentation for more details.


SHPC Condo Slurm Resource Queues

This page describes Slurm resources queues in the CADES SHPC Condo environment. The hardware in each queue, queue access policies, and batch directives required to access each queue are described. If your access group is not listed here, please check the Moab resources queues section.

The following tables below list the technical specifications of each resource queue available in the CADES SHPC Condos. For more information on submitting jobs to a resource queue, view the Execute a Slurm Job page. The SBATCH directives required to submit to each queue are listed in the SBATCH directive section near the bottom of the page.

Open Slurm Condo Queues

Name # Nodes Cores Micro arch. RAM Max Walltime (D-H:M:S) Local Scratch GPU
testing 4 32 Broadwell 62G 4:0:0 160G N/A
batch 169, 44 32, 36 Haswell, Broadwell 125G 14-0:0:0 250G, 2T N/A
skylake 49 36 Skylake 187G 14-0:0:0 2T N/A
high_mem 98 36 Broadwell 250G 14-0:0:0 2T N/A
high_mem_cd 14 36 Skylake 375G 14-0:0:0 2T N/A
high_mem_d 2 32 Haswell 500G 14-0:0:0 2T N/A
gpu 12 32 Haswell 250G 14-0:0:0 250G 2x K80 (GK210)
gpu_p100 8 36 Broadwell 500G 14-0:0:0 2T 2x P100 (GP100)

Burst Queue (All nodes except testing queue)

Name Max Walltime (D-H:M:S) # Nodes
burst 2-0:0:0 381

📝 Note: Jobs run in the burst queue can be preempted at any time.

To set preempted jobs to restart automatically add the following to your batch script:

#SBATCH --requeue

Resource Queue Access

Groups which have purchased nodes in a resource queue have priority access to those resources when submitting jobs. Groups which have not purchase resources in a queue can still submit jobs to the queue through the burst directives, but their jobs will be preempted and restarted if a user with priority submits a job that needs the resources. This is why we note that all jobs run in the burst queue can be preempted.

Resource queue group access quick reference:

Purchased Queue Resources Can Submit Jobs Job Flags
Yes Yes Preemptor
No Yes Preemptee, Restartable

To submit a job with priority access to your group resources, you need to specify the queue that those resources are in in your job script. The queues that each group has priority access to are listed in the 'Nodes by Group' tables below. You can also determine which queues you have priority access to with the sinfo command.

To submit a job to resources your group does not have priority access to, you need to specify the burst queue. The burst queue contain all nodes from all queues and allows you to schedule preemptable jobs on any free resources in the cluster. Details on the burst queue are shown in the queues table above.

Open Condo Slurm Nodes by Group

Group Nodes
birthright gpu:10, high_mem_cd:18
bsd batch:25, high_mem:12, high_mem_cd:8, gpu:3
ccsd batch:25, high_mem:20, gpu_p100:8
ccsi batch:30
chem high_mem_cd:6
cnms batch:38, high_mem:32
mstd high_mem:2
ntrc high_mem:32
nucphys high_mem_d:2
sns batch:50,skylake:5
theory batch:30,skylake:22
qmc skylake:22

Resource Queue SBATCH Directives

The SBATCH directives required to submit jobs to each resource queue are listed below organized by group. Each group has two lists of directives:

  • Standard SBATCH directives
    • These directives are used to submit non-burst jobs.
  • Burst SBATCH directives
    • These directives are used to submit burst jobs.

📝 Note: This syntax indicates that you should pick one of the options in brackets. Lines without brackets can be copied without any changes.

  • [ option_a | option_b | ... ]

Group List:

Please note: Groups that are still on the PBS partition are listed in thier own own section.

📝 Note: If you do not see your group listed, please contact the CADES team and include:

  • Help with SHPC Condo Registration
  • UCAMS ID or XCAMS ID, contact information, reason for requesting an SHPC Condo allocation, and the name of your directorate and division.

Resource Queue Directives by Group

CADES Birthright

Standard SBATCH directives:

#SBATCH -A birthright
#SBATCH -p gpu

Burst SBATCH directives:

#SBATCH -A birthright
#SBATCH -p burst

Biosciences Division (BSD)

Standard SBATCH directives:

#SBATCH -A bsd
#SBATCH -p [batch|high_mem|high_mem_cd|gpu]

Burst SBATCH directives:

#SBATCH -A bsd
#SBATCH -p burst

Computing and Computational Sciences Directorate (CCSD)

Standard SBATCH directives:

#SBATCH -A ccsd
#SBATCH -p [batch|high_mem|gpu_p100]

Burst SBATCH directives:

#SBATCH -A ccsd
#SBATCH -p burst

Climate Change Science Institute (CCSI)

Standard SBATCH directives:

#SBATCH -A ccsi
#SBATCH -p batch

Burst SBATCH directives:

#SBATCH -A ccsi
#SBATCH -p burst

Chemical Sciences Division (CSD)

Standard SBATCH directives:

#SBATCH -A chem
#SBATCH -p high_mem_cd

Burst SBATCH directives:

#SBATCH -A chem
#SBATCH -p burst

Center for Nanophase Materials Sciences (CNMS)

Standard SBATCH directives:

#SBATCH -A cnms
#SBATCH -p [batch|high_mem]

Burst SBATCH directives:

#SBATCH -A cnms
#SBATCH -p burst

Materials Science and Technology Division (MSTD)

Standard SBATCH directives:

#SBATCH -A mstd
#SBATCH -p high_mem

Burst SBATCH directives:

#SBATCH -A mstd
#SBATCH -p burst

National Transportation Research Center (NTRC)

Standard SBATCH directives:

#SBATCH -A ntrc
#SBATCH -p high_mem

Burst SBATCH directives:

#SBATCH -A ntrc
#SBATCH -p burst

Nuclear Physics (nucphys)

Standard SBATCH directives:

#SBATCH -A nucphys
#SBATCH -p high_mem_d

Burst SBATCH directives:

#SBATCH -A nucphys
#SBATCH -p burst

Spallation Neutron Source (SNS)

Standard SBATCH directives:

#SBATCH -A sns
#SBATCH -p [batch|skylake]

Burst SBATCH directives:

#SBATCH -A sns
#SBATCH -p burst

MSTD First-Principles-Based Materials Theory (MSTD Theory)

Standard SBATCH directives:

#SBATCH -A theory
#SBATCH -p [batch|skylake]

Burst SBATCH directives:

#SBATCH -A theory
#SBATCH -p burst

MSTD Quantum Monte Carlo (MSTD QMC)

Standard SBATCH directives:

#SBATCH -A qmc
#SBATCH -p skylake

Burst SBATCH directives:

#SBATCH -A qmc
#SBATCH -p burst

ARM Slurm Testing Directives

Slurm is available in the CADES Open SHPC Condo through the load-balanced login nodes or-slurm-login.ornl.gov. These nodes can be ssh'd into directly from the ORNL network:

$ ssh <uid>@or-slurm-login.ornl.gov

Use the following group specific batch directives in your #SBATCH script. Full batch script examples can be found in Execute a Slurm Job.

#SBATCH -A arm
#SBATCH -p testing

Moderate Condo Slurm Testing Directives

Slurm is available in the mod-condo through the login nodes mod-slurm-login01.ornl.gov and mod-slurm-login02.ornl.gov. These two nodes are accessible by first logging into one of the existing mod-condo login nodes, and then the Slurm login node:

$ ssh <uid>@mod-condo-login01.ornl.gov
$ ssh <uid>@mod-slurm-login01.ornl.gov
`

Use the following group specific batch directives in your #SBATCH script. Full batch script examples can be found in Execute a Slurm Job.

NSED:
#SBATCH -A nsed
#SBATCH -p testing
NSET:
#SBATCH -A nset
#SBATCH -p test

results matching ""

    No results matching ""