External (untracked) datasetsMarch 1, 2020 at 9:49pm
I see you've mentioned this capability as under development in other posts but just thought I'd register interest here. We deal with a lot of very large (multi TB-scale) datasets and datasets on network drives so having an easy of interacting with these in Gigantum would be great!
March 2, 2020 at 4:30pm
- thanks for the feedback. We're hearing this a lot recently and will prioritize a better solution soon. We do have a small change coming out very soon (probably this week) that adds an
untrackedfolder to the
inputsection of a project. Once this is in place, you could use symlinks on you host to mount directories on your host or on network drives into a project (we'll be adding some tips on how to do this in our documentation). The data would not be versioned, and other users would have symlink in the same way to collaborate, but it'll work.
To collect a little extra info for future development, what kind of network shares do you use? If we supported NFS as a dataset type would that work for you? Do you care if the network drive is mounted to the host as well as the Project container? How do you control access to your network drive data sources? Any info is super helpful! If you don't want to post publicly, DM me. thanks!
March 3, 2020 at 12:52am
- awesome, that will work nicely to start with! We have a mix of NFS and Windows File Server at the moment but think we're going to transition everything to Dell Isilon 1FS (?) soon The shares would be mounted to both the host and the container Access is controlled via our Microsoft AD accounts
March 25, 2020 at 2:31pm
Cool. We've done some initial testing, and mounting NFS shares (tested on macOS and Linux) works for mounting into the container. On Linux, you can use bind-mounts on the host for any arbitrary data. See here for details: https://docs.gigantum.com/docs/untracked-folders#section-mounting-external-data-via-untracked-folders
It might work on Windows, but we haven't had an opportunity to dig into arbitrary mountpoints (I've only mounted NFS volumes as the equivalent of another drive on Windows). If you make progress with that, let me know!