Eva Dziadula with students at the Universidad Popular Autónoma del Estado de Puebla (UPAEP) in Puebla, Mexico

Global migration is accelerating and bringing with it new challenges. One Notre Dame economist is designing digital tools to help students, scholars, leaders, and nations visualize migration and prepare for it more effectively.

Researchers often study the flow of things like rivers, electricity, or blood. Eva Dziadula studies the flow of something very different: people. A teaching professor in the University of Notre Dame's Department of Economics, Dziadula works to identify and understand patterns in migration. Read more.

A visualization of the immigration patterns of highly skilled workers. Darker shades of green indicate higher levels of immigration. Hashed texture indicates incomplete data.

Biannual Maintenance
Friday, May 19 – Sunday, May 21, 2023

The Center for Research Computing (CRC) would like to announce that all computational and storage systems will be offline as they undergo maintenance during this time.

When the CRC began planning to refresh our shared compute capabilities in the fall of 2022, we anticipated adding around 100 new servers.  By the time all was said and done, additional faculty interest pushed the final purchase to more than 200 servers.  We soon realized that this would be a challenge to implement within our existing infrastructure, not only in terms of floor space for server racks, but also with both power and cooling.  We began discussions with our facilities management who quickly came through with a plan to expand our current footprint.  With a new server room, we'll be able to support an additional 17 server racks and will also have access to improved power options.  This is the first time the CRC data center has expanded in nearly 10 years and should provide us space to grow for many more.  The current plan is to finish populating the expansion during our May maintenance weekend.

I/O (Input/Output) access is essential for any program that needs to read, write, or list files. However, excessive I/O access can cause performance issues, especially in shared environments where multiple users are accessing the same storage device simultaneously. High I/O traffic can lead to delays, slow down other users' tasks, and even crash the file system. In this article, we will explore how memory mapping can help reduce I/O load when working with large files.

Memory mapping is a technique that allows a portion of a file to be mapped directly into memory, making it accessible as if it were already part of the program's memory. This eliminates the need for the program to repeatedly read data from the file on disk, which can be slow and time-consuming. By using memory mapping, the program can read and write to the mapped region in memory, significantly reducing the number of disk reads and writes required. This can lead to improved performance and less strain on the file system.

To illustrate how memory mapping works in practice, let's look at an example in Python:

import mmap
# Open a file and memory map it
with open('large_file.txt', 'rb') as f:

with mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) as mm:

# Read the contents of the file into memory
contents = mm.read()

# Process the contents of the file
# ...

In this example, we open the file "large_file.txt" and memory map it using the mmap function. We then read the entire contents of the file into memory in a single operation using the read method of the memory-mapped file object (mm). After reading the contents of the file into memory, we can then process the data as needed.

It's important to note that memory mapping may not always be the best approach for working with files. For smaller files, it's generally simpler and more efficient to use Python's built-in file I/O functions, like open(), read(), and write(). Additionally, memory mapping may require a significant amount of memory, so it's important to only use it when necessary and to be mindful of memory usage.

In summary, memory mapping is a powerful technique for reducing I/O load when dealing with large files in computer programs. By mapping files directly into memory, programs can read and write data to files more efficiently, avoiding the need for multiple I/O operations and improving performance. While it may not be suitable for all file sizes and types, memory mapping is an essential tool for managing I/O load in high-performance computing environments.

Every Wednesday and Thursday in May
2:00 – 3:00 p.m.
Flanner Hall, CRC Training Room 812 (map)

This training is available to new users and current users interested in a refresher course on how to use CRC resources. Attendees learn the basics of accessing CRC resources and submitting jobs on the CRC clusters. This course is a co-requisite when receiving a CRC account. Learn more.

June 26-30, 2023
Santa Fe, New Mexico

ACM/IEEE Joint Conference on Digital Libraries (JCDL) will be held on June 26-30, 2023, at the Los Alamos National Laboratory in Santa Fe, New Mexico. CI Compass, the NSF Cyberinfrastructure Center of Excellence, will host a workshop during the conference, with Don Brower, Research Assistant Professor and Computational Scientist, and Co-Organanizer of the CI Compass FAIR (Findable, Accessible, Interoperable, and Reusable) Topical Working Group. The workshop will include sessions on the Nelson Memo and Federal Data Management Guidelines, Outcomes of the CI Compass FAIR Survey on FAIR Strategy and Implementation, and a panel discussion on Challenges with Data Management at major facilities. Learn more.

June 27-30, 2023
Washington, D.C.

The National Science Foundation (NSF) Research Infrastructure Workshop will be held June 27 - 30, 2023, in Washington, D.C. On Tuesday, June 27, CI Compass will host its Cyberinfrastructure Workshop. Jarek Nabrzyski, Director of the Center for Research Computing, will host a panel titled “A Discussion on Different Models of Data Governance,” and Charles Vardeman, Research Assistant Professor and Computational Scientist, will host a panel titled “The Expanding Use of Artificial Intelligence (AI) in Research Infrastructure applications.” Learn more.

531,079 CPU hours
Physics

312,313 CPU hours
Applied & Computational Mathematics & Statistics

288,610 CPU hours
Civil & Environmental Engineering & Earth Sciences

276,697 CPU hours
Aerospace & Mechanical Engineering

259,469 CPU hours
Aerospace &  Mechanical Engineering

250,121 CPU hours
Civil & Environmental Engineering & Earth Sciences

228,479 CPU hours
Aerospace & Mechanical Engineering

203,391 CPU hours
Chemical & Biomolecular Engineering

202,755 CPU hours
Biology

174,028 CPU hours
Aerospace & Mechanical Engineering