Considering Four gets his mind messed with at least once per season, this is all fine for him!

Just be aware that while Four is the prominent life/mind/what have you, there are three previous incarnation's memories in his mind. They are his memories, but The Doctor in those memories will not look like him.

Also any memories with The Third Doctor might be a bit messed up. Canonly, Three's memories of Two and One were really messed up. Any questions, feel free to ask! I can clear it all up xD