He knows that if he so much as looks at Sansa wrong, Brienne will likely take his hand off (if he's lucky). The idea of having to watch his actions even in his own home is irksome, but if the way to get Sansa to come is to allow the others as well, it's a price he's willing to pay.

"As soon as it's light enough to see the road." That way they'll have as much daylight as possible for the actual travel.