When you use the graphical tool to remove Full Access to a mailbox, it removes your permissions to the mailbox, but it remains in your profile. And just to make it interesting, you can't remove it from Outlook. You'll get an error when you attempt to delete it and it's not visible in the Outlook profile.
The error in outlook when attempting to close the leftover mailbox:
This group of folders is associated with an e-mail account. To remove the account, click the File Tag, and on the Info tab, click Account Settings. Select the e-mail account, and then click Remove.
The solution is to use the Exchange Management Shell to remove the permissions instead. When you do so, it removes the additional permission that adds the account to your profile.
Remove-MailboxPermission MbxName -User UserWithPermissions -AccessRights Fullaccess