The paint left in the grain will show, but depending on its colour may not be all that noticeable.
You could try something like a brass bristle brush if you don't want to hit it with paint stripper.
My recommended finish - seeing as you don't want a high gloss - would be an oil finish; one of the decking oils would be an easy way to find something.
No matter what finish you use it will need maintennance - if you go the varnish route you'll be looking at sand and revarnish around every 5 years, which I think is considerably more work than quickly wiping on some fresh oil every 6-12 months.
I'd probably be tempted to use an epoxy resin finish on the top and bottom of the door to seal it well in those 'out of sight, out of mind' areas.