You don't need to take the middle tree, but it would be highly recommended to at least dual cast the spells. With just Kindred Mage and Animage you get a +10 or +8 bonus to your spells. Calm affects up to level 9, and with Kindred Mage, that goes to level 19. If you dual cast that, it affects up to level 42. For reference, a Bandit Chief is level 28. With Pacify, you can hit humans up to level 30 with a single cast.
If you also take the middle tree, you'll get +8 levels to your calm spell, bringing the single hand cast to level 27. As soon as you get to Pacify, you can hit humans up to level 38 with a single hand cast.
For an assassin character, you can use Muffle and Invisibility in one hand and a dagger in the other. Or you can start by hitting everyone with an area effect dual cast Pacify, then switch to Invisibility and dagger.
If you are using Destruction magic also, the Aspect of Terror perk from the middle tree gives +10 damage to all fire spells.
Illusion is a very powerful tree, I say you just stick with the right side only, and see where you have problems, decide whether or not you want to take the dual casting perk for select situations, or if you'd rather take some of the middle tree, or neither!
