The CRC is very excited to announce that we are expanding our storage infrastructure and providing new options this spring. Taking user feedback from our storage survey earlier this year, we set out to find a file system that would bring performance and reliability, affordable storage tier options, and backup security. After meeting with multiple providers and performing a cost-benefit analysis, we chose to partner with NetApp going forward. Our initial investment will add more than 3 petabytes (PB) of usable capacity. The new appliances have arrived, have been installed across multiple sites, and internal validation is currently underway. External beta testing is expected to open up within the next month or so, so be on the lookout for invitations.

Tier 1 of the new system will host user and group directories. This tier is backed by all-flash SSD and NVME drives. We expect that many users will notice a performance benefit over current AFS and Panasas offerings. All tier 1 folders will also keep the traditional nightly snapshot and offsite backup as is currently done on AFS. Tier 2 will provide an economical storage solution for infrequently used data, with the option to add backup as needed. We are increasing the amount of free storage for faculty to 5 terabytes (TB) and will be providing additional capacity at cost. We are also implementing a new design layout which will emphasize group space over personal home directory storage. The motivation behind this is to improve data locality and reusability, reduce redundant file copies, and improve data management processes.

So what does this mean for AFS? In the short term, we will continue to run AFS alongside the new NetApp and both systems will be usable. We're currently projecting that we will announce the retirement of AFS in mid-2025 with a 12- to 18-month migration period built in. During that time, we will be offering training opportunities for data stewardship as well as assisting both individuals and large groups with data movement between the systems. More information to come soon.

Bash, the Unix shell, is a powerful command-line interface that allows users to perform a wide variety of file manipulation tasks. Whether you're coding or processing large amounts of data, mastering file manipulation commands in Bash can significantly enhance your efficiency. Below is a brief  guide covering the most commonly used file manipulation commands in Bash.

1. Viewing File Content

cat file.txt: Displays full content (useful for small files).

less file.txt: Scrolls through large files (q to quit).

head -n 20 file.txt / tail -n 20 file.txt: View first/last lines.

tail -f output.txt: Follow file in real-time (useful for logs).

2. Creating and Editing Files

nano file.txt / vi file.txt: Opens text editors for file editing.

3. Copying Files

cp source.txt destination.txt: Copies a file.

cp -r /source_dir /destination_dir: Copies entire directories.

cp -p source.txt destination.txt: Copies with file attributes.

4. Renaming or Moving Files

mv oldname.txt newname.txt: Renames files.

mv file.txt /destination_dir/: Moves files.

5. Deleting Files

rm file.txt: Deletes a file.

rm -r directory/: Deletes a directory.

rm -i file.txt: Interactive prompt before deletion.

6. Searching Within Files

grep "search_term" file.txt: Finds patterns in files.

grep -r "search_term" /directory/: Searches recursively.

7. Redirecting Output to Files

echo "Hello, World!" > output.txt: Overwrites file.

echo "New Line" >> output.txt: Appends to file.

command > output.txt 2>&1: Redirects output and errors.

8. File Permissions

chmod +x script.sh: Grants execute permission.

9. File Compression and Archiving

tar -cvf archive.tar file1.txt: Creates archive.

tar -xvf archive.tar: Extracts archive.

gzip file.txt / gunzip file.txt.gz: Compresses and decompresses files.

10. Checking Disk Usage

du -sh /directory: Disk usage of a directory.

du -h file.txt: Disk usage of a file.

Conclusion

Bash offers versatile file manipulation commands that streamline workflows. Mastering these basics can significantly boost productivity for users managing code or data sets.

Every Wednesday and Thursday
2:00 – 3:30 p.m.
812 Flanner Hall (map)

The CRC offers multiple training opportunities for both new and existing users. We periodically provide short courses and other learning opportunities, which are advertised on our website and through email lists. In-person office hours are held every Wednesday and Thursday from 2:00-3:30 p.m. in Flanner Hall, room 812, on a first-come, first-served basis. You can also arrange a Zoom meeting at your convenience by emailing CRCsupport@nd.edu with your availability. We recommend bringing a laptop to in-person sessions.

For more information, please contact CRCsupport@nd.edu.

Aerospace & Mechanical Engineering
756,973 CPU hours

Physics
586,793 CPU hours

Chemical & Biomolecular Engineering
468,187 CPU hours

Chemical & Biomolecular Engineering
389,198 CPU hours

Chemistry & Biochemistry
356,404 CPU hours

Chemical & Biomolecular Engineering
248,797 CPU hours

Chemical & Biomolecular Engineering
235,116 CPU hours

Physics
231,687 CPU hours

Chemical & Biomolecular Engineering
230,129 CPU hours

Biological Sciences
219,890 CPU hours