Hi Matt,
Thanks for your update.
As you said:"The larger account(currently 4gb) kept throwing me the Error: RemoteServerIsSlowException: Data migration for this mailbox failed because of delays on the remote server."., this issue could be caused by throttling control/connection limit of the source IMAP server or firewall.
Regarding:"I have swapped the dns so that I can start using exchange now, I guess I can try a full migration still if you think it would help.", since you have swapped the DNS, I suggest you try a full migration.
If you need further assistance, please feel free to let me know.
Regards,
Jerry