Orchestrating and embedding MutagenJuly 15, 2019 at 4:43am (Edited 11 months ago)
July 19, 2019 at 9:39am
Hi , have a question about how mutagen behaves in case of multiple instances.
For example, we have mutagen executable not system-wide, but in our project's scope. We've using it to create sync sessions exclusively with our docker containers.
How the sync sessions will behave if the system has system-wide mutagen already installed (via homebrew for example). If there was also active other sync sessions started with that system mutagen?
Basically, we are thinking about
scopedmutagen instance because we will need to control versioning and we don't want to touch system configuration (for cases if it is used in other projects).
Thanks in advance!
July 20, 2019 at 6:28am
Hey , at the moment there's no "scoping" based on executable - everything is attached to the single user-global Mutagen daemon. The closest thing to scoping in v0.9.x is using labels to scope/select sessions, but I think what you're asking for is the ability to maintain a separate Mutagen daemon on a per-project basis, correct?
What if I were to add an envionment variable like
MUTAGEN_DATA_DIRECTORYthat tells Mutagen to look somewhere other than
~/.mutagenfor the daemon socket and data? You could then do:
MUTAGEN_DATA_DIRECTORY=/some/directory mutagen daemon start
Or, if you just want to host the process yourself:
MUTAGEN_DATA_DIRECTORY=/some/directory mutagen daemon run
Then when you script commands, it would look like:
MUTAGEN_DATA_DIRECTORY=/some/directory mutagen sync create ...
Would that work? I'd prefer to do this as an environment variable since (a) it would be a little experimental and (b) it avoids needing to add
--data-directoryflags to every command.
The agent binaries and data on remote systems would still go into
~/.mutagenon those systems, but that's fine since that data is designed to co-exist with multiple clients.
July 23, 2019 at 2:29pm
Thanks for the informative reply !
ability to maintain a separate Mutagen daemon on a per-project basis, correct?
Yep. In our case, our toolset will manage many projects which will be controlled by sync session labels. The idea was to have a separate mutagen executable in the scope of our toolset.
A configurable environment variable is a good option, all commands will be scripted anyway.
Will it allow us to start/stop mutagen daemon separately from another, possibly, active mutagen instance? And control only sync sessions which were created by "our" mutagen instance. So for example command
mutagen daemon stopwon't break "system" mutagen sessions and daemon.
mutagen daemon runcommand is not documented yet, I never heard about it before. :)
July 25, 2019 at 9:25am
Hey , I've just tagged v0.10.0-beta2, which allows the type of embedding that you want to do. The documentation for v0.10.x is going out tomorrow on the new website (stay tuned), but there's a quick summary of the changes in the release notes for v0.10.0-beta1.
Anyway, what you can now do is set the
MUTAGEN_DATA_DIRECTORYenvironment variable to an absolute path that will serve as the data directory and IPC endpoint for an isolated daemon instance. It won't have any effect on other Mutagen daemon instances. When it comes to embedding, the best way to do it is with the aforementioned
mutagen daemon runcommand, which is a hidden command that serves as the entrypoint for the daemon (i.e. it doesn't background). So your toolset would do something like:
MUTAGEN_DATA_DIRECTORY=/some/internal/directory mutagen daemon run
Of course, you can still use the normal daemon
stopcommands to create a backgrounded daemon instance, but if you're embedding the daemon, this is probably easier to control. To communicate with that daemon, you just specify the same environment variable value to other Mutagen commands, e.g.
MUTAGEN_DATA_DIRECTORY=/some/internal/directory mutagen sync create ...
MUTAGEN_DATA_DIRECTORY=/some/internal/directory mutagen daemon stop
One thing worth noting about the v0.10.x release is that the commands have been restructured a little bit. Since Mutagen's functionality is expanding, the synchronization session commands now live under
mutagen sync. However, to keep scripts from breaking, I've left the original
create/list/flush/pause/resume/terminateavailable as hidden aliases under the root command, so
mutagen createand the other commands still work.
If you get a chance to play with v0.10.0-beta2, I'd really appreciate any feedback. There's a lot of new stuff. I wouldn't ship it out or announce it to your users yet, but hopefully it won't be long until the final release is tagged. I'd like to make that happen this week.
August 20, 2019 at 3:51pm
I'm integrating mutagen v0.10.0 to our project and everything great and works smoothly so far.
We are using the new orchestration feature and faced with small limitation, it's not possible to run flush command for the exact session with
mutagen project flushcommand. Maybe there is another way to flush separate session in project scope? Or it's better to name sessions with project prefix and use
mutagen sync flush <session_name>?
August 20, 2019 at 9:58pm
Hey ! The synchronization (and forwarding) sessions created by the orchestration infrastructure are still normal sessions, they just have an additional
io.mutagen.projectlabel attached to them. This means that they can still be flushed using the
mutagen sync flushcommand by using their session ID (or by name, though names may not be unique). But I'm thinking now that maybe it would be helpful to add label support to the project configuration? I.e. if you could assign labels for each session in the
mutagen.ymlfile, you could more precisely specify them using the
mutagen sync flushcommand. Would that be helpful?
Basically, labels are already assigned in the project's mutagen.yml, like
sync:labelName1:alpha: ...beta: ......labelName2:...
In our case, if we reuse mutagen.yml for different projects,
mutagen sync flush labelName1will affect all active projects sync sessions with name labelName1? I think command like
mutagen project flush labelName1would be good to flush only labelName1 session of the specific project.
So the keys below
sync:are actually session "names" rather than labels, but you're correct that they aren't unique and that using
mutagen sync flush <name>may very well affect sessions outside of the project. I was proposing adding a
labelskey to the session configuration, but actually your solution is better. Internally, the project orchestration can map something like
mutagen project flush <name>to a
<name> && io.mutagen.project=<project-id>requirement. This will take a bit of work, but it is more elegant. I'll try to draft up a proposal for that and get back to you.