Firstly, check that the link pipe is inserted all the way into the silencer, it’s a tight fit and quite easy not to get fully home.
Secondly, and probably what will help you most, is don’t line things up by the spring loops welded to the link pipe and silencer. If you try and get them in line, the exhaust will only ever be in one position, you won’t be able to adjust anything; the springs work perfectly well when fitted at an angle (and to be truthful they aren’t really need when everything is snugged up). Now rotate the link pipe in the silencer (with everything loosely fitted) anticlockwise looking from the rear of the bike. Every degree you turn the link pipe anticlockwise will result in the link pipe’s exit (and the silencer attached to it) moving outwards and downwards. This, I reckon, will give you the room you’re looking for?
Although I mounted my hanger in a different position to that recommended, the silencer is still in exactly the same position as it would be if I did use the pannier frame top mount. You can see from the pictures below that the springs are twisted by maybe 20-25 degrees (as a result of the anticlockwise twisting I mentioned). The second image shows the link pipe’s bends viewed directly along the line of the chainguard, which should give you an idea of routing.
What I will add though, is bear in mind that the "X-Black" silencer is shorter and narrower than both the round and oval versions, although the link pipe is the same.