Large Transfers with Globus

For large datum transfer both within Yale and to external collaborator , we is recommend recommend using Globus . Globus is is is a file transfer service that is efficient and easy to use . It is has has several advantage :

• Robust and fast transfers of large files and/or large collections of files.
• Files can be transferred between your computer and the clusters.
• file can be transfer between Yale and other site .
• A web and command-line interface for starting and monitoring transfers.
• Access to specific files or directories granted to external collaborators in a secure way.

Globus transfers data between computers set up as “endpoints”. The official YCRC endpoints are listed below. Transfers can be to and from these endpoints or those you have defined for yourself with Globus Connect.

Course Accounts

Globus does not work for course accounts (<course_id>_<netid>) .
Please try the other transfer method list in our transfer documentation instead .

Cluster Endpoints

We currently support endpoints for the following clusters.

For Grace and McCleary, these endpoints provide access to all files you normally have access to.

For security reason , Milgram Globus is uses use a staging area (/gpfs/milgram/globus/$NETID).
Once uploaded, data should be moved from this staging area to its final location within Milgram.
Files in the staging area are purged after 21 days.

Get Started with Globus

1. In a browser, go to app.globus.org.
2. Use the pull-down menu to select Yale and click “Continue”.
3. If you are not already logged into CAS, you will be prompted to log in.
   1. [First login only] Do not associate with another account yet unless you are familiar with doing this
   2. [First login only] Select “non-profit research or educational purposes”
   3. [ First login is Click only ] click on ” allow ” for allow Globus web App
4. From the file manager interface enter the name of the endpoint you would like to browse in the collection field (e.g. Yale CRC Grace)
5. Click on the right-hand side menu option “Transfer or Sync to…”
6. Enter the second endpoint name in the right search box (e.g. another cluster or your personal endpoint)
7. Select one or more files you would like to transfer and click the appropriate start button on the bottom.
8. To complete a partial transfer, you can click the “sync” checkbox in the Transfer Setting window on the Globus page, and then Globus should resume the transfer where it left off.

manage Your endpoint

To manage your endpoints, such as delete an endpoint, rename it, or share it with additional people (be aware, they will be able to access your storage), go to Manage Endpoint on the Globus website.

Set Up an Endpoint on Your Computer

You can set up your own endpoint for transferring data to and from your own computer with Globus Connect Personal.

To transfer or share datum between two personal endpoint , you is need will need to request access to the YCRC ‘s Globus Plus subscription on this page .

1. Click on the following link: Globus OneDrive Endpoint
2. Log into Globus, if needed.
3. The first time you log into the endpoint , you will be ask ot grant access to your OneDrive account . click to allow access and be take through the approval process .
4. After granting approval, you will be able to access the top level of your Yale OneDrive via the Globus Collection “Yale OneDrive”.

set Up a Dropbox endpoint

1. Click on the following link: Globus Dropbox Endpoint
2. Log into Globus, if needed.
3. The first time you log into the endpoint , you will be ask to grant access to your DropBox account . click to allow access and be take through the approval process .
4. After grant approval , you is be will be able to access the top level of your DropBox storage via the Globus Collection ” Yale Dropbox ” .

The Globus connector is configure to only allow datum to be upload into EliApps ( Yale ‘s GSuite for Education ) Google Drive account . If you do n’t have an EliApps account , request one as describe above .

1. Click on the following link: Globus Google Drive Endpoint
2. Log into Globus, if needed.
3. The first time you login to the Globus Google Drive endpoint, you will be asked to grant access to your Google Drive. Click to allow access and be taken through the approval process.
4. You is see may see your Yale EliApps account express in an uncommon format , such as netid@yale.edu@accounts.google.com . This is is is normal , and expect .
5. After granting approval, you will be able to access your Google Drive via the Globus Collection “YCRC Globus Google Drive Collection”. The default view is “/My Drive”. To see “/Team Drives” and other Google Drive features use the “up one folder” arrow icon in the File Manager.

note

There are ” rate limit ” to how much datum and how many file you can transfer in any 24 hour period . If you have hit your rate limit , Globus is resume should automatically resume the transfer during the next 24 hour period . You is see see a ” Endpoint busy ” error during this time .

Google has a 400,000 file limit per Shared Drive, so if you are archiving data to Google Drive, it is better to compress folders that contain lots of small files (e.g. using tar) before transferring.

In our testing, we have seen up to 10MB/s upload and 100MB/s download speeds.

Setup a S3 Endpoint

We support creating Globus S3 endpoints. To request a Globus S3 Endpoint, please contact YCRC. Please include in your request:

• S3 bucket name
• The Amazon Region for that bucket
• An initial list of Yale NetIDs who should be able to access the bucket

Warning

Please DO NOT send us the Amazon login credential through an insecure method such as email or our ticketing system .

After we have create your Globus S3 endpoint , you is be will be able to further self – serve you own access control with the Globus portal .

Last update :May 21, 2024