MRI Data Storage and Access

Where is Data Stored?

MRI Data

After each session, MRI image data and Siemens physio data from the scanner end up in three places:

Data is maintained on the scanner console computer for approximately 2 weeks. This is the average amount of time before it needs to be deleted to allow room for new data.
Data is backed up locally onto external hard drives. Although not a reliable long term storage method, this allows us to maintain a local, easy-to-access archive of all scan data within the MRI Facility.
Data is sent to the Rotman Research Institute NeuroImaging Database (RRINiD), an XNAT archive server maintained by RIT. This is both our most secure long term data storage and our method of data retrieval for researchers both on and off site.

Ancillary Equipment Data

Other data types from secondary systems are currently managed directly by the researchers. For example, any files generated by your stimulus scripts, or data collected from EEG, Eye-Tracking, or Biopac data are only stored locally on their respective systems. It is the responsibility of the researcher to transfer these data via appropriate methods depending on the ethical nature of the data. This can involve secure media such as encrypted USB devices, or secure transfer methods like SFTP transfers to Rotman servers, where systems allow. There are long term plans to centralize other data types on RRINiD, but no timelines for implementation.

RRINiD (XNAT)

Data can be retrieved from RRINiD (XNAT) directly by researchers. Users requesting XNAT access must be listed on the relevant REB approved Personnel Log. Next, submit an RIT Ticket (Type: Web Services – RRINiD (XNat) new account or connection problem) requesting account creation for the user with access to the particular MRI Project Number. You should get password instructions from RIT when the ticket has been completed.

RRINiD Website

RRINiD (XNAT) can be accessed in a web browser here:
https://rrinid.rotman-baycrest.on.ca/spred/

On RRINiD (XNAT) you will have access to all MRI Projects that you belong to. XNAT uses a project – subject – session hierarchy that you can explore to find your MRI image data and your Siemens physio data, if applicable. You can use the “Manage Files” button (NOT the “Download” button) in the toolbar on the right side of any Session page to download any combination of scans from a single session in a zip file.

Jacob’s RRINID script

Jacob has been developing a script to do a variety of data download and management tasks with XNAT from the command line. The script is still in a development state and is subject to significant changes in future versions but is working smoothly. Please reach out to Jacob Matthews to request a copy of this script.

Here is the documentation for an overview of what the script is designed to do:

RRINID <subfunction> <subfunction arguments>
This script allows for simple command line interaction with RRINID (Rotman XNAT), including:
-listing the data available at different levels of RRINID
-downloading experiments from RRINID individually or in batch
-converting experiments into Nifti and/or BIDS directories

You can browse RRINID in your browser here:
https://rrinid.rotman-baycrest.on.ca/spred/app/template/Login.vm

Common Examples:
  RRINID list 102 14771
   -Show all experiments available for project 102 and subject 14771
  RRINID download /data/ 102 14771 MRI_20230527
   -Download and sort all MRI scans for project 102, subject 14771, experiment date 20230527
  RRINID convert /data/ 102 14771
   -Download and convert all experiments for project 102, subject 14771

The script will use your current server account username as your RRINID username.
Alternate usernames are not accepted to enforce data ownership aligns with REB.
The script will prompt you for your RRINID password.

Input Glossary:
<localdatadir> is a file path for the local data directory where the output will be created.
  It’s recommended to use the same path for all calls to this function.
  Project directories will be created within this directory.
  Within project directories will be subdirectories for each data type (dcm, nii, bids, physio).
  Data will be organized by subject and experiment within these directories.
  BIDS project structures will be automatically managed within the bids folder by HeuDiConv.
  Data Structure Example:
   localdatadir/
   ├─ projectid/
      ├─ bids/
         ├─ PIName
            ├─ RAName
               ├─ BIDSProjectDir
      ├─ dcm/
         ├─ subjectid
            ├─experimentid
              ├─ scan.dcm
      ├─ nii/
         ├─ subjectid
            ├─experimentid
              ├─ scan.nii.gz
      ├─ physio/
         ├─ subjectid
            ├─experimentid
              ├─ PhysioSource
                 ├─ physio.ext/card/resp
<projectid> is the Project ID in a long XNAT format or short MRI project number format
  (e.g. For Jean Chen MRI 102, Long=’ChJe_M102_BA’ Short=’102′)
  This script will attempt to match a given Short Project ID to its Long Project ID.
  If this fails, please supply the Long Project ID, which can be found on the RRINID website.
<subjectid> is the Rotman Subject ID (e.g. ‘6537’ or ‘14985’)
<experimentid> is the Experiment ID (e.g. ‘MRI_20150325’, ‘MRI_20150325_b’, ‘PhysioData_20150325’)

Sub-Functions:

list
  RRINID list <projectid> <opt:subjectid> <opt:experimentid>
   The list subfunction will list all data within the level you specify.
    i.e. all subjects within a specified project
         all experiments within a specified project and subject
         all scans/items within a specified project, subject, and experiment

download
  RRINID download <localdatadir> <projectid> <opt:subjectid> <opt:experimentid>
   The download subfunction will download and organize all data within the level you specify.
    i.e. all subjects within a specified project
         all experiments within a specified project and subject
         all scans/items within a specified project, subject, and experiment
   MRI data will be organized within <localdatadir>/projectid/dcm/
   Physio data will be organized within <localdatadir>/projectid/physio/
   Some known problem-causing data types will be automatically detected and removed.
    This includes: MPR scans, non-ND paired scans, duplicate ND scans, and Siemens environment files.
   RRINID download will skip already downloaded data, so it is safe to make repeat calls.

convert
  RRINID convert <localdatadir> <projectid> <opt:subjectid> <opt:experimentid>
   The convert subfunction will download and convert all data within the level you specify.
    i.e. all subjects within a specified project
         all experiments within a specified project and subject
         all scans/items within a specified project, subject, and experiment
   Data will be converted as follows:
    Data compatible with HeuDiConv using ReproIn conventions (default at RRI MRI since 2022)…
     …will be converted and organized within <localdatadir>/projectid/bids by HeuDiConv.
     This includes scans names starting with anat, func, fmap, and dwi.
    Data that is known to only function as dicoms will be left in <localdatadir>/projectid/dcm.
     This includes spectroscopy scans.
    The remaining data will be converted to Nifti+json format with dcm2niix…
    …and will be left in <localdatadir>/projectid/nii.
     This includes asl, perfusion, and other less common scans.
    RRINID convert can be run after RRINID download and will use the existing downloaded data.
    RRINID convert will skip already converted data, so it is safe to make repeat calls.

delete
  RRINID delete <localdatadir> <projectid> <opt:subjectid> <opt:experimentid>
   The delete subfunction will delete all data within the level you specify.
    i.e. all subjects within a specified project
         all experiments within a specified project and subject
         all scans/items within a specified project, subject, and experiment
    This subfunction will delete data from dcm, nii, and physio sub-directories.
    This subfunction is unable to act on data within <localdatadir>/bids.

Curl Commands

If you would like to script your own download tools, the most direct way to do so is to use curl commands. These same curl commands are the core of Jacob’s RRINID script. Here are some examples of curl commands used to interact with different data types on RRINiD. The overall structure of the commands is largely similar. Understanding the URL structure XNAT uses is what differentiates most of these. Anywhere you see words in all caps surrounded by braces, you are meant to substitute the identified information. The curl commands below may contain newline characters for formatting purposes. You may need to ctrl-shift-v paste them or otherwise remove the formatting or newline characters for them to work properly.

1) How do I find out what PROJECT ID’s are on RRINID?

>curl -k –u {USERNAME}:{PASSWORD} -X GET –-url https://rrinid.rotman-baycrest.on.ca/spred/data/projects?format=csv

URL Notes:
-No levels are supplied with specific labels, and projects is the last category listed, so all project ids will be returned.
-format=csv specifies that we are returning a list of information, not any actual data.

2) How do I find out the RRINiD projects that I have access to?

There does not appear to be a way to do this with a simple curl command. A large script would be required. Visit the RRINiD website and the home page will only list projects you have access to.

3) How do I find out what SUBJECT ID’s are in a PROJECT?

curl -k -u {USERNAME}:{PASSWORD} -X GET –url https://rrinid.rotman-baycrest.on.ca/spred/data/projects/{PROJECTID}/
subjects?format=csv

URL Notes:
-All levels up to projects are supplied with specific labels, and subjects is the last category listed, so all subject ids will be returned.
-format=csv specifies that we are returning a list of information, not any actual data.

4) How do I find out what EXPERIMENT ID’s are in a SUBJECT?

curl -k -u {USERNAME}:{PASSWORD} -X GET –url https://rrinid.rotman-baycrest.on.ca/spred/data/projects/{PROJECTID}/
subjects/{PROJECTID}_{SUBJECTID}/experiments?format=csv

URL Notes:
-All levels up to subjects are supplied with specific labels, and experiments is the last category listed, so all experiment ids will be returned.
-format=csv specifies that we are returning a list of information, not any actual data.

5) How do I find out what SCAN ID’s are in an EXPERIMENT?

curl -k -u {USERNAME}:{PASSWORD} -X GET –url https://rrinid.rotman-baycrest.on.ca/spred/data/projects/{PROJECTID}/
subjects/{PROJECTID}_{SUBJECTID}/experiments/
{PROJECTID}_{SUBJECTID}_{EXPERIMENTID}/
scans?format=csv

URL Notes:
-All levels up to experiments are supplied with specific labels, and scans is the last category listed, so all scan ids will be returned.
-format=csv specifies that we are returning a list of information, not any actual data.

6) How do I download a single SCAN ID’s data from a specific EXPERIMENT?

curl -k -u {USERNAME}:{PASSWORD} -X GET -o {PATH/FILENAME.zip} –url https://rrinid.rotman-baycrest.on.ca/spred/data/projects/{PROJECTID}/
subjects/{PROJECTID}_{SUBJECTID}/experiments/
{PROJECTID}_{SUBJECTID}_{EXPERIMENTID}/
scans/{SCANID}/resources/DICOM/files?format=zip

URL Notes:
-All levels up to scans are supplied with specific labels, and files is the last category listed, so all files within that scan id will be returned.
-resources has already been specified as DICOM. This is to differentiate from the SNAPSHOT dicom sample slice images that XNAT automatically generates and stores in the same structure.
-format=zip specifies that we are downloading the actual data in a zip file.

7) How do I download multiple SCAN ID’s data from a specific EXPERIMENT?

curl -k -u {USERNAME}:{PASSWORD} -X GET -o {PATH/FILENAME.zip} –url https://rrinid.rotman-baycrest.on.ca/spred/data/projects/{PROJECTID}/
subjects/{PROJECTID}_{SUBJECTID}/experiments/
{PROJECTID}_{SUBJECTID}_{EXPERIMENTID}/
scans/{SCANID},{ SCANID},{SCANID}/resources/
DICOM/files?format=zip

URL Notes:
-All levels up to scans are supplied with specific labels, scans is supplied multiple comma separated scan ids, and files is the last category listed, so all files within each listed scan id will be returned.
-resources has already been specified as DICOM. This is to differentiate from the SNAPSHOT dicom sample slice images that XNAT automatically generates and stores in the same structure.
-format=zip specifies that we are downloading the actual data in a zip file.

8) How do I download all SCAN ID’s data from a specific EXPERIMENT?

curl -k -u {USERNAME}:{PASSWORD} -X GET -o {PATH/FILENAME.zip} –url https://rrinid.rotman-baycrest.on.ca/spred/data/projects/{PROJECTID}/
subjects/{PROJECTID}_{SUBJECTID}/experiments/
{PROJECTID}_{SUBJECTID}_{EXPERIMENTID}/
scans/ALL/resources/DICOM/files?format=zip

URL Notes:
-All levels up to experiments are supplied with specific labels, scans is supplied with the ALL tag, and files is the last category listed, so all files within all scan ids will be returned.
-resources has already been specified as DICOM. This is to differentiate from the SNAPSHOT dicom sample slice images that XNAT automatically generates and stores in the same structure.
-format=zip specifies that we are downloading the actual data in a zip file.

9) How do I download all SCAN ID’s data from all EXPERIMENTS from a specific SUBJECT?

curl -k -u {USERNAME}:{PASSWORD} -X GET -o {PATH/FILENAME.zip} –url https://rrinid.rotman-baycrest.on.ca/spred/data/projects/{PROJECTID}/
subjects/{PROJECTID}_{SUBJECTID}/experiments/
ALL/scans/ALL/resources/DICOM/files?format=zip

URL Notes:
-All levels up to subjects are supplied with specific labels, experiments and scans are supplied with the ALL tag, and files is the last category listed, so all files within all experiment ids and all scan ids will be returned.
-resources has already been specified as DICOM. This is to differentiate from the SNAPSHOT dicom sample slice images that XNAT automatically generates and stores in the same structure.
-format=zip specifies that we are downloading the actual data in a zip file.

10) How can I download a Subject physio data?

curl -k -u {USERNAME}:{PASSWORD} -X GET -o {PATH/FILENAME.zip} –url https://rrinid.rotman-baycrest.on.ca/spred/data/projects/{PROJECTID}/
subjects/{PROJECTID}_{SUBJECTID}/experiments/
{PROJECTID}_{SUBJECTID}_{EXPERIMENTID}/
scans/ALL/resources/DICOM/files?format=zip

URL Notes:
-Only the naming of the experiment id differentiates physio data (e.g. “PhysioData_19991231”) from mri data (e.g. “MRI_19991231”). All physio files are uploaded as a single zip file under a scan id labeled “PhysioFiles”. You could therefore specify this scan id as well, but it isn’t necessary.
-All levels up to experiments are supplied with specific labels, scans is supplied with the ALL tag, and files is the last category listed, so all files within all scan ids will be returned.
-resources has already been specified as DICOM. This is to differentiate from the SNAPSHOT dicom sample slice images that XNAT automatically generates and stores in the same structure.
-format=zip specifies that we are downloading the actual data in a zip file.