I don't think common knowledge of the Trek universe (recognition of "Beam me up, Scotty!" and the like aren't common knowledge of the universe) is pervasive enough in our culture today to make a show almost exclusively about aliens in Trek successful. You could probably get a movie out of it, but not a series. Any series is going to have to be Federation-centric to be successful, though it doesn't have to have a majority of human cast members. Just a couple here and there, and enough aliens that are human-like enough to identify with (and that falls under personality more than appearance).
Personally, I think that a new Trek series isn't going to have any more challenges being set in the existing timeline/canon than a completely new series with its own cannon. The problem isn't having trouble coming up with new ideas in the existing canon, it's coming up with fresh and solid stories, period. Hollywood has always been known to be more about flash and action than story, and that reputation has only been growing as better-than-real-life CGI effects have gotten better and easier to shovel in. Getting a good story is a challenge, regardless of what universe it's set in, and a show like Trek is going to require, first and foremost, a good story. It doesn't matter if it's NuTrek or Ye' Olde Trek, if the story and writing and dialogue are crap, the show just won't work. It's not the kind of show that can get away with crap writing and crap ideas and stories and still pull in solid ratings. And getting good, relevant stories that engage people is going to be a challenge for any Trek, Nu or Ye Olde, because all the 'low-hanging fruit' stories have already mostly been done, by Trek or some other show, and what's left will quickly be exhausted. Writing a good Trek show, regardless of which canon it uses, is going to require real effort in writing and storytelling, something that the Hollywood culture has largely been shying away from.
