Not entirely sure what way the wall is out of plumb. The unit needs to be dead square and plumb/level in its opening by using packers against the framing. The unit does not need to be dead plumb sideways though as you can really only keep them in line with the walls. It will perform better but 6mm out of plumb sideways is not a problem really providing the unit is twisted.
I have modified Corinthian units prior to install by cutting down either the framed unit side (closing side) including track or just the closing side track. Once it was necessary to cut 20 mm from closing side track as the door I was using was only 800mm wide. It would have had a 10mm gap at the back when closed otherwise. Another occasion I'd cut a unit down both closing and opening sides/tracks to allow a door to go back flush with the closing reveals.
What brand and size unit are you using? as there are many brands and a lot I've probably never used or seen.