Cloud Computing Expansion in Scientific Research

In recent years, major cloud service providers like Google Cloud, Amazon Web Services, and Microsoft Azure have significantly expanded their offerings for scientific computing. These platforms now provide access to:

Our Progress with Cloud Integration

We're excited to share that our beta program with Google Colab has shown promising results, allowing many of you to leverage powerful computing resources without the constraints of local hardware limitations. Building on this success, we're exploring native integration with additional cloud-based tools and services to complement our existing on-premises infrastructure.

We Need Your Input

As we evaluate which additional cloud services to support, we'd like to hear directly from you:

Please respond via the following survey link here.

How Your Feedback Will Help

Your input will directly inform our resource allocation decisions and help ensure that our computational infrastructure evolves to meet your changing research needs. We aim to provide a balanced ecosystem of local and cloud resources that best serves our diverse scientific community.

When developing Python scripts, there are times you may need to run external programs or scripts—like R or shell commands—as part of your workflow. Python’s built-in subprocess module is a powerful tool for doing just that. It allows you to launch new processes, interact with their input/output streams, and capture their return codes.

This guide walks you through using subprocess to run an external R script from within Python.

1. Import the Required Modules

Start by importing the necessary modules:

import subprocess

import os

The os module ensures that the subprocess inherits your current environment settings.

2. Initialize the Environment Modules System

The CRC system uses environment modules to manage software environments. You can initialize them with the following command:

exec(open(f'{os.environ["MODULESHOME"]}/init/python.py').read())

This sets up the module command so you can load software modules directly from Python.

3. Load the Required Module

To load a specific software module, such as R version 4.4.0:

module("load", "R/4.4.0")

This makes the R executable available in your current environment.

4. Run the External Program

To execute an R script (e.g., my_script.R), use subprocess.run() like this:

subprocess.run(["R", "CMD", "BATCH", "my_script.R"])

Here’s what each part of the list means:

Summary

Python’s subprocess module makes it easy to run external programs from your scripts. By initializing your environment properly and using subprocess.run(), you can seamlessly integrate tools like R into your Python-based workflows. 

Civil & Environmental Engineering & Earth Sciences
1,064,109 CPU hours

Chemical & Biomolecular Engineering
553,557 CPU hours

Aerospace & Mechanical Engineering
442,111 CPU hours

Civil & Environmental Engineering & Earth Sciences 
 410,763 CPU hours

Chemical & Biomolecular Engineering
329,849 CPU hours

Chemical & Biomolecular Engineering 
267,032 CPU hours

Civil & Environmental Engineering & Earth Sciences
217,743 CPU hours

Physics
207,988 CPU hours

Chemical & Biomolecular Engineering
193,951 CPU hours

Chemical & Biomolecular Engineering
164,409 CPU hours

Chemical & Biomolecular Engineering
24,595 GPU hours

Chemical & Biomolecular Engineering
13,694 GPU hours

Computer Science and Engineering
9,414 GPU hours

Computer Science & Engineering
9,325 GPU hours

Computer Science and Engineering 
 7,717 GPU hours

Computer Science and Engineering
6,730 GPU hours

Computer Science & Engineering
5,906 GPU hours

Chemical & Biomolecular Engineering
4,605 GPU hours

Chemistry & Biochemistry 
4,432 GPU hours

Chemistry & Biochemistry
3,962 GPU hours

Every Wednesday and Thursday
2:00 – 3:00 p.m.
812 Flanner Hall (map)

The CRC offers multiple training opportunities for both new and existing users. We periodically provide short courses and other learning opportunities, which are advertised on our website and through email lists. In-person office hours are held every Wednesday and Thursday from 2:00-3:30 p.m. in Flanner Hall, room 812, on a first-come, first-served basis. You can also arrange a Zoom meeting at your convenience by emailing CRCsupport@nd.edu with your availability. We recommend bringing a laptop to in-person sessions.

For more information, please contact CRCsupport@nd.edu.