To simplify the transfer of files into the lakehouse, we should... add a destination database for the lakehouse!
The lakehouse stores data in S3 buckets, so we'll make use of the Globus S3 connector for this. Every user that logs into the lakehouse has a set of individual S3 credentials that permit Globus to move data in and out of these buckets. Inside its private network, KBase provides a service ("MMS") that accepts an ORCID for a user and returns that user's S3 credentials. The DTS will likely be moved inside the KBase network so it can interact with MMS.
When the DTS has a user's S3 credentials, it has to somehow communicate these credentials to Globus. "Ordinarily" this is done by a Globus user, who logs in and sets up their S3 credentials manually, but we don't want to ask users to do this. Ideally we would have a programmatic way to add S3 credentials for a Globus user, but it doesn't look like this is possible from the relevant documentation. And I can certainly understand why one wouldn't want to automate something like this.
Maybe we can insert some logic (like a command-line call in the right environment) to add an S3 credential, triggered when a user logs into the lakehouse hub (if such a credential doesn't already exist).
To simplify the transfer of files into the lakehouse, we should... add a destination database for the lakehouse!
The lakehouse stores data in S3 buckets, so we'll make use of the Globus S3 connector for this. Every user that logs into the lakehouse has a set of individual S3 credentials that permit Globus to move data in and out of these buckets. Inside its private network, KBase provides a service ("MMS") that accepts an ORCID for a user and returns that user's S3 credentials. The DTS will likely be moved inside the KBase network so it can interact with MMS.
When the DTS has a user's S3 credentials, it has to somehow communicate these credentials to Globus. "Ordinarily" this is done by a Globus user, who logs in and sets up their S3 credentials manually, but we don't want to ask users to do this. Ideally we would have a programmatic way to add S3 credentials for a Globus user, but it doesn't look like this is possible from the relevant documentation. And I can certainly understand why one wouldn't want to automate something like this.
Maybe we can insert some logic (like a command-line call in the right environment) to add an S3 credential, triggered when a user logs into the lakehouse hub (if such a credential doesn't already exist).