USCMS Computing Intern (Data Management LHC-CMS)

Position Description

The CMS “Data Management” team is looking for a highly motivated and skilled DevOps and AI Engineer to join our team. In this role, you will play a crucial role in developing, operating and evolving the data management system of the Compact Muon Solenoid (CMS). You will also help with a chatbot with RAG or contribute to setting up more artificial intelligence (AI) operations.

Come help us manage more than half an Exabyte of tape storage, 250 PB of disk, and about half a million CPU cores, provided by various data centers, including High Performance Computing. We support the central operations of CMS in both data management and in operational AI. We are looking for one Research Intern employed by UW-Madison but based at Fermilab to operate the CMS transfer system. On average, about 10 PB is transferred weekly between all CMS sites. The successful candidate would monitor the system, maintain its health, and debug transfer problems together with site admins and transfer system developers. The successful candidate would ensure smooth operation by developing and optimizing tools and monitoring transfers using the RUCIO data management system. The successful candidate would receive training and ongoing technical advice and would work with development teams across other parts of the CMS computing infrastructure. Interns can also work with an Operational AI team on an LLM-based chatbot with RAG capabilities, and potentially on more complex AI/ML methods such as training anomaly detection algorithms.

CMS Program Description

The Compact Muon Solenoid (CMS) is an experiment that investigates a wide range of physics using the Large Hadron Collider (LHC) at CERN in Geneva, Switzerland. The CMS collaboration uses a large and distributed computing infrastructure to record, process, and analyze CMS data and simulation at petabyte scales. Centrally, we ensure that data processing jobs are run correctly and their output is safely stored. CMS Computing Operations manages these central workflows, starting with processing freshly recorded data at CERN, re-processing data, and producing Monte Carlo simulations at worldwide distributed computing centers. CMS Computing Operations also maintains and manages large-scale data movements between these different centers and collaborates with the admins of the more than 70 computing sites across the world to maintain and sustain functional services and resources used by central workflows and physicists performing analysis of detector data and simulations.

Computing systems used at and between the different sites are based on Grid technologies and use sophisticated software to achieve the required high scales of transfer volumes and processing throughput. State-of-the-art programming technologies are used for the different systems. The processing infrastructure is based on distributed state machines that handle job submission and tracking. The data transfer system is based on a highly performant node computing infrastructure both centrally and at the individual computing centers. All systems use the latest database technologies, both local and distributed, including relational and NoSQL database systems. Central services use a hierarchical HTTP cache infrastructure to provide access to alignment and calibration constants of the detector to all jobs running on the Grid. To check site performance, a sophisticated test suite regularly probes the different services of each Grid site and enables early discovery and mitigation of problems. Uninterrupted service across all sites and central services is essential to deliver centrally produced samples to the CMS collaboration and to guarantee analysis access for all physicists to produce high-quality physics results. Expert operators are needed to continually monitor all central workflows, transfers, and site performance because most workflows are time-critical and essential to the overall success of the CMS experiment. A good overview is shown in the following animation: https://cds.cern.ch/record/1541893?ln=en

Our team has built a specialized chat bot designed to consolidate information across diverse sources, including Twiki and JIRA, structuring this data inside an open knowledge graph. Currently hosted on a Kubernetes cluster, this platform sees active use by CMS Computing Operations. Over the last year, the initiative has made significant progress, and we are now seeking a Research Intern to help refine the service, incorporate user feedback, and broaden its utility, accessibility, and strategic roadmap.

Technical Requirements:

• Experience with Python and bash scripting

Experience with Linux and shells

• Experience with software development and code design using git

(Basic)

• Experience with SQL databases (Oracle knowledge is a plus)
• Knowledge of batch systems and mass storage systems is recommended
• Knowledge of GRID infrastructure and GRID technologies is recommended
• Experience with monitoring tools like Prometheus, Grafana, and OpenSearch
• Testing and troubleshooting distributed systems
• Working fluency with AI coding assistants and agents (expert level not required): knowing when they help, giving them good context, and checking what they produce

Behavioral Requirements:

-
Good Communications Skills

• Working knowledge of English, both oral and written
• Ability to work independently
• Demonstrating flexibility: adapting quickly and resourcefully to shifting priorities and requirements.
• Working in teams: working well in groups and readily fitting into a team; participating fully and taking an active role in team activities.
• Learning and sharing knowledge: keeping up-to-date with developments in your own field of expertise and readily absorbing new information.
• Organizational skills

Details:

This is an opening for a term Research Intern hired through the University of Wisconsin-Madison (UW-Madison) and stationed at the Fermi National Accelerator Laboratory in Batavia, Illinois, USA. The successful candidate is required to have a Bachelor's degree in engineering, computing, mathematics or natural sciences. The stay is limited to 1 year, with the possibility of a second-year extension.

The successful candidate is provided with:

• Salary as determined by experience/qualifications
• Benefits (e.g., health, dental, and vision) are included.
• Paid legal holidays as determined by UW-Madison.
• The UW-Madison visa office provides assistance to acquire a visa, if needed.
• Reasonable accommodations for qualified individuals with disabilities.

The University of Wisconsin is an Equal Opportunity Employer. It fulfills its public mission by creating a welcoming and inclusive community for people from every background - people who, as students, faculty, and staff, serve Wisconsin and the world.