The show makes it seem that Mary didn't see her child from the time he was very, very young. In fact she left him and Scotland when he was about a year old.
When Darnley is walking through the woods with James you can see what appears to be Will Kemps real life wedding ring while he's holding the baby.