background mask refinement

I do a fair amount of background editing and while the background mask tool is a great start (I shoot mostly headshots/portraits with solid color - grey - background), I have to do a fair amount of cleanup of the mask. The refine mask tool is too much of a "sledgehammer", and the magic brush is hit or miss. What would be great is to have options for refining the mask based on a color range, proximity/area of the picture. For example, if I person on a grey background, I have to clean up some areas of the skin which get masked - it would be nice to be able to erase any portion of the mask that is in the color range of the skin tone (or not have it be masked in the first place).