From my father, is almost his immediate reply. While it's true that Ned Stark has been a father to him, and will always be like a father to him in many ways, the man in front of him now is also his father, the one responsible for his existence. Even if this is the only time they ever talk, he has to reconcile the fact that due to some twist of fate, he has two fathers.

He nods at the bartender and waits until he gets his drink and the man is no longer paying attention to them to answer. "From the man who raised me. Ned Stark." Now he's revealed not only that he knows Lyanna's full name but also how he knows it, and why he looks familiar, too. He's a Stark. "Lyanna Stark was my mother. She told him that before she died."