The actor, 52, who played Lord Grantham alongside Elizabeth McGovern as his wife Cora, said he is already beginning to feel “nostalgic” after the hit drama ended last December.
Hugh said working alongside Dame Maggie, who played the Dowager Countess and recently starred in acclaimed film The Lady in the Van, made him raise his game.
Speaking on Radio 4’s Desert Island Discs, he said: “I can remember the very first scene I did with her and I was absolutely terrified, and I think I can remember the last scene with her and I was absolutely terrified.
“She is the most astonishing actress. Her wit is legendary, as you say, and she doesn’t suffer fools. And you raise your game, you have to. It’s great acting opposite her because she’s got extremely high standards for herself and expects them of others.”
He added: “I think I’m already sort of nostalgic about it. It was an extraordinary rollercoaster of a show, to be involved with something that you thought was going to last six or seven episodes, as we did in the first series, and then 52 episodes later we’ve said goodbye.
“And those people involved – I think there were about sort of 40 or 50 people who’d done every episode, been both in front of the camera and behind – and the hundreds of crew who’d come and gone over the six years.
“It does become a second family. So the final farewells were sad, but obviously the freedom … normally I’d be gearing up to don the thermals and start shooting again in Highclere Castle, so it’s a strange feeling not to be doing that. But happiness tinged with fond memories.”