To all chip-cpu users,

A cooling issue in a research park computer room necessitated the unscheduled and immediate shutdown of chip-cpu hardware which was housed in that computer room. While running jobs were inevitably lost, the decision to quickly shutdown the hardware was made to preserve the hardware. Maintenance technicians are currently assessing the chillers in the computer rooms and DoIT staff are on site to assist.

The GPU hardware associated with chip-gpu is unaffected.

The newly acquired storage system we've recently migrated most research volumes to is also unaffected.

Thank you for your patience as we navigate this issue. Please expect an update no later than Friday (Jan 16) morning or as the issue is resolved.

---

Roy Prouty

Asst Director for Research Computing
UMBC DoIT
Live, from Research Park Computer Room

Hi everyone,

First I want to congratulate everyone for making it to the end of the semester, and a special congratulations to our new graduates! Second, happy holidays! I hope whatever you do with your break it's relaxing and fulfilling! 

I'm writing to give everyone an update regarding the UMBC HPCF and other Research Computing projects currently undertaken by DoIT.

Summary

• Staff will be continuing to reach out to PIs in an effort to migrate storage volumes to the new RRStor system

• Scheduled cluster downtime

• SIG UE Nominations 

Migration of Storage to RRStor

At this time, we have completed almost all migrations for PI specific groups. Non-PI storage, such as those for datasets, ongoing grants, and classes are ongoing, and are expected to be completed by mid- to late-January. Please continue to work with HPC staff to get these scheduled and migrated!

Cluster Manager Updates

On Thursday Dec 18, 2025 we had a scheduled downtime to patch our cluster management software in order to improve stability and memory management. There should be no impact on the user experience as a result of these changes.

SIG UE Nominations
In an effort to gain a consistent set of users to come to the SIG UE meetings, we have started a SIG UE nomination process. Selected members will be expected to come to as many of the SIG UE meetings as possible, work with the HPC user base to identify problems, and relay this information to the HPC staff. If you're interested in being selected as a member please respond to this Google Form by January 14th, 2026: https://docs.google.com/forms/d/e/1FAIpQLSeRmu5ixHGCoMv-vwFfRMMfe2uBRQ5puwSEqUaZLj0D3w6hDw/viewform?usp=preview .

Publications
If you have any publications, presentations, theses, or other works that made use of the campus cluster(s), please submit an RT Ticket with bibliographic information so that we can accurately reflect this work in our records and on the HPCF Website. 

Need Help?
As always, please communicate any issues/questions to the Research Computing RT Queue (hpcf.umbc.edu > User Support > Request Help).

The User Experience Committee will be meeting next week, Tuesday November 18th 1330-1430 in room 401 of the Physics Building, to have its second meeting of the Fall semester. We encourage all users to attend, especially non-PI users (including but not limited to: undergraduate students, graduate students, post-doctoral students, etc.)

What is the User Experience Committee?

The User Experience Committee is a new committee created by the Shared Infrastructure Governance group dedicated to addressing training, support, and accessibility of research computing infrastructure at UMBC. Beyond discussion of structured user support and experience questions, the SIG User Experience Committee will provide an open forum for users to share insights, identify areas for improvement, and help guide enhancements that best support the needs of the computational research community.

Steps to Address Feedback of Last Meeting

Based on feedback from our last meeting, we are improving communication and usability for Chip users. We have more clearly announced the new SIG User Experience committee and are refactoring relevant wiki pages for better clarity. We are planning an HPC Workshop Series for the Spring semester and will begin posting reminders before office hours. We've also improved the RT form by adding wiki links and are working with the RT team to streamline PI group selection. Posting AI tooling costs publicly is not permitted by DoIT.

If you have feedback you would like to give PLEASE join us at next week's SIG User Experience Meeting!

Hi everyone,

Today (October 8th 2025) at around 14:30ET, we experienced an unexpected system downtime. While investigating, we found that the failure stemmed from a failure on the file server associated with the ada GPU cluster (ada-rstor). In order to make the chip hardware available to users, we have temporarily disabled all network connections between the chip cluster and this ada-rstor file server.

Please know that we are currently working to understand the underlying issue with the ada-rstor file server. We hope to be able to make the ada-rstor volumes available ASAP. 

While we know this action disproportionately affects users of the chip-gpu cluster and we are appreciative of users' patience as we navigate this issue.

In the meantime, the chip cluster compute hardware along with any volumes should be available to all users. If you notice any issues with the use of chip unrelated to ada-rstor volumes (file paths starting with "/umbc/ada"), please report them via RT with as much information as possible.

Max Breitmeyer

HPC Specialist

Hi everyone,

I'm writing to give everyone an update regarding the UMBC HPCF and other Research Computing projects currently undertaken by DoIT.

Summary

• Office hours! We have them! They are available via appointment at https://hpcf.umbc.edu/help/office-hours/ or drop-in in-person on Fridays at 1:30 in Engineering Building Room 102
• We've added some new GPU partitions in an effort to make it more clear what nodes are "safe" from preemption
• Staff will be continuing to reach out to PIs in an effort to continue migrating to the new RRStor system
• SIG meetings updates and new "User Support" SIG group 
• SSH connection issues

Office Hours
In coordination with the ScaleS center, we are excited to announce that drop-in office hours have returned for this academic semester. They will be held weekly on Fridays at 1:30PM in Engineering Building Room 102. In fact, we've already held our first drop-in office hours on September 19th and it was a SMASHING success.

When coming with questions please note that it is a first come, first served system, and to have as much information ready as possible. Remember, the faster we can get started on your issue, the faster we can fix it. We will continue to offer one-on-one appointments for more in-depth problems or if the drop-in hours don't work for you.

You can see a listing of all drop-in office hour times on the HPCF myUMBC group events listing: https://my3.my.umbc.edu/groups/hpcf/events 

For scheduling an office hour appointment, please refer to this page: https://hpcf.umbc.edu/help/office-hours/ 

GPU Partitions
During our last downtime, one of our main goals was to implement a new GPU Slurm model, where contributing PIs have priority on machines that they helped to purchase.

In an effort to relieve some of the confusion around these changes, we've added two new partitions gpu-general and gpu-contrib. The gpu-general partition will place a job on a node that is not owned by one of the contributing PIs. This guarantees that the job will run for at least three days without being preempted when this partition is specified in the Slurm commands. The gpu-contrib partition is not a usable partition, meaning that no one can submit jobs to this partition. This partition shows users what nodes are owned by one (or more) PIs. 

If you don't care what node you are placed on you can still use the standard gpu partition. 

Migration of Storage Volumes
While the new Ceph Storage Cluster (RRStor) is here to stay, the Isilon storage system will be sunset at the end of this calendar year. The Isilon storage system was purchased at the start of the COVID-19 Pandemic, but its proprietary file system, high cost of expansion, and high software support costs make it no longer feasible for us to maintain.

Instead, we'll be taking a portion of the saved expense from the Isilon and dedicating it to expanding RRStor and its capabilities. This will require that we migrate research storage volumes from the Isilon Storage System to Ceph before the end of the calendar year. This represents the careful movement of more than 2PiB of data, so DoIT Research Computing staff will be working with researchers to schedule the migration of their storage volumes so as to minimize impact on the research enterprise. Note that the duration of each volume migration will depend on the size of the volume.

Staff will continue reaching out to PIs to schedule migrations. At this time, we have about 10% of all migrations completed. We ask that you continue to work with us to make sure that we have all volumes scheduled by October 15th, and sunset the Isilon volumes completely by Thanksgiving.

SIG
The SIG-CPU and SIG-GPU Committees have met and selected members. Please see the Shared Infrastructure Governance webpage for more information and past meeting materials. The following SIG Committees have meetings scheduled, and will be visible on this myUMBC Group Events page.

• The CPU Committee will hold a meeting of voting members October 2, 2025 
• The GPU Committee will be meeting October 27, 2025 at 13:30ET, all researchers using GPUs are welcome. Location TBD
• The Cloud Committee is working to identify membership and meeting times 
• The User Experience Committee will be meeting on the Third Tuesdays of each month at 13:30ET. These user experience meetings are open to any user of the chip HPC. Location TBD

SSH Connection Issues
We've seen an increase in the number of users saying they are having connection issues. Specifically with repeated closures of their active SSH connections. We're currently working alongside our DoIT network engineering team to determine the cause of the issue. Any information about a dropped connection you might have experienced will help us. The information we need to know is the following:

• Date and time of login
• Date and time of dropped connection
• Are you on/off campus?
• Are you on a wired/or wireless network?
• Are you using the VPN?
• Which login node did you connect to initially (chip-login1/2)?

Publications
If you have any publications, presentations, theses, or other works that made use of the campus cluster(s), please submit an RT Ticket with bibliographic information so that we can accurately reflect this work in our records and on the HPCF Website. 

Need Help?
As always, please communicate any issues/questions to the Research Computing RT Queue (hpcf.umbc.edu > User Support > Request Help).

Thanks for reading,
Roy Prouty
Assistant Director for Research Computing