Since 18 November 2026, Slurm on HORUS holds every job to the memory it requests. If you don’t ask for memory, your job gets the partition’s default per CPU, which may be less than it needs.
Why we made the change
On 18 November, a few jobs that asked for 62.5 GB each used up to twice that. Three compute nodes ran out of memory, and the jobs of everyone else on those nodes were interrupted. With the new limit, a job that uses far more than it asked for is stopped on its own, and the node and the other jobs keep running.
What to do
#SBATCH --mem=8G for the whole job on each node, or #SBATCH --mem-per-cpu=4G.lc and lm, 3,000 MB in lg and 4,000 MB in batch.OUT_OF_MEMORY.sacct -j <jobid> -o JobID,ReqMem,MaxRSS,State. Ask for somewhat more than the peak.The SLURM page lists the partitions, their memory defaults and their limits.
Questions, or a job that was stopped and you’re not sure why: horus-help@umich.edu.
Thank you, Shawn McKee for the OSiRIS and HORUS projects
Tags