It's possible I just don't know of a way, but once you are on the Strange Island, you can only walk to Enpolad's garden. And yes there are non-magical ways to Enpolad's garden, but they require you to SAY or YELL a certain phrase. Both ways that I'm aware of will not work for those with minospeech.

So if you're playing a rager minotaur, avoid strange island. Unless, like I say, there's a nonmagical way out of there that I'm unaware of.