No mention is made of retconning Rhys. Rhys not only met Adam (although not properly introduced), but he would have remembered Gwen's memory problems during those 48 missing hours that the Torchwood team had to forget, not to mention the fact that she threatened to kill him.
Adam tells Jack that if Jack chooses to forget Adam, he will not only lose the memory that Adam made himself a part of, but that he would lose all memory of his father. "Your father will cease to have existed." Why would losing a single day's memory wipe out all memories of Jack's father? And, obviously, Jack still remembered later how he lost his brother.