Embed Size (px)
Citation preview
Imperial College London3 Million jobs a year and counting …
Simon Burbidge 16 Sep 15
Imperial College London
• Premier UK University and research institution ranked #2= (with Cambridge) in QS World University rankings (MIT #1)
• #9 in worldwide Universities (THES) and #3 in Europe• 15,000 students, 6,200 of them postgrad• 7,200 staff (2,600 of which dedicated to research)• Science based – engineering/technology/medicine• Lots of industrial research collaboration• £350M per year income
Main Campus South Kensington London
• Located on the site of the 1851 Great Exhibition in London
HPC Service
• University wide service• £2M annual budget for hardware and software• Additional income directly from research groups• 5 dedicated HPC staff• Largest UK University HPC system (HPC-SIG)
3 HPC systems
• cx1 – low end cluster mostly ethernet• cx2 – high end parallel MPI cluster (SGI ICE)• ax3/4 – large shared memory dedicated to genomics (UV)
ax3/4
•Large memory SMP•Two SGI UV- 384 processors 5 TB memory- 1024 processors 16 TB memory•Main user is Bioscience – Next Generation Gene Sequencing •PBS topology aware scheduling, cpusets
•cx2
•High end MPP commissioned Autumn 09, upgraded twice since, about to be refreshed, supporting large highly parallel jobs•SGI Altix ICE 8400 EX + ICE 7000 cores•Dedicated to large MPI parallel jobs•PBS topology aware scheduling – simple config, serial and small parallel workload
cx1
• Low end cluster• Mostly gigabit ethernet, some infiniband islands• Upgrading all the time, parts of the system owned by particular
research groups• 24178 cores, 1468 nodes, 13 distinct hardware types plus GPUs• Tricky to schedule!
•PBSPro
•In use on all 3 systems•3 separate PBS servers, one for each system type•Different PBS configuration on each system
• PBSPro – cx1
• Largest system in terms of cpus• non-uniform topology• Many node types/sizes• Complex scheduling decisions required• All scheduling decisions resource based – users only use queue name
for private queues• Fair share very important• Stability very important – 80,000 + jobs per day on average – that’s
30M jobs a year!• Use node grouping extensively
How’s it set up?
• Resource based queue selection – route only queues, except for those owned privately.
• Users specify walltime , number of nodes and node features (cpus, memory, infiniband)
• Nodes are not assigned to queues, instead node grouping is used to maintain fauxqueue as well as other groupings (eg switch connectivity)
How’s it used?
• Serial jobs and Serial array jobs• Single node parallel jobs• Multiple node parallel jobs• - Essentially each category is scheduled separately
So that’s all good then…
• My perfect solution keeps everyone happy and the researchers highly productive
• But ….. Soon a few cracks appear• mostly created by the users !
Why isn’t my job running?
• Answer – because its fair share and you are too greedy• Or, your resource specification is out of range and there are no nodes
which can run your job• “But my job still isn’t running, I can see other peoples jobs jobs running,
why not mine? Its not FAIR!”• Well it is fair (that’s why its called fair share right?)• Solution – stop users from being able to view other peoples’ jobs – what
the users can’t see they can’t complain about.• Except qstat can give away some stuff in the queue definitions.• Also, it seems fair that owners of private queues should be able to see
the work on their queue.
Why isn’t my jobs running? I’M AN IMPORTANT PERSON!
• Of course all my carefully set up limits and controls are very good and successful, but there’s always an exception – the VIPs and the deserving needy
• Solution – set specific per_user limits and controls as well as generic ones. We change these quite often in response to particular user needs and this works very well
• NB : remember to take off the special limit once the user’s special need is over.
Over enthusiastic users
• As the system is so easy to use and beautifully set up, some users are tempted to run vast numbers of jobs.
• Server and scheduler can clog up• Helps to set queue max_jobs to avoid catastrophe (but deleting the
excess jobs can be painful).
Over enthusiastic users – 2
• With so many jobs run, users sometimes don’t check them.• Or, it didn’t work so I’m going to re-submit them all again• Put scientist in front of computer and scientific method flies out of the
window!• Users need to check the results, just like in real experiments, something
might have gone wrong.• Creation of tools to check large quantities is useful
But I ran it 3 times!
• And the results were wrong every time!• The users need to not only check the results, but to validate them too.
• PBSPRo – futures ?
• With more and more cores per server, we are beginning to see bottlenecks in PBS performance.
• We’d like to see even more complex scheduling expressions, eg. compound expressions “and”/”or”.
• Also to be able to schedule on the basis of what’s inside the select directive.
