The exchanged mails between the IMAP host and the MTA need a unique identifier to organize contents of the DB, and this would not be possible or automatic if your switched the upstream MTA.

It sure is possible. I’ve copied maildirs over different software, different servers, local copies back to the server and so on. Also if you just rely on your own IMAP server the upstream doesn’t matter as fetchmail (or whatever you choose to use) anyways communicates between hosts on their preferred protocols.

Obviously there’s a tradeoff since now you’re responsible for your backups and maintaining your server, but it can sit nicely on your private LAN with access only locally or via VPN without direct access to the internet. And you don’t need MTA to run IMAP server in the first place.

You can run a dovecot (or any IMAP server) where ever you want and use fetchmail to pull data from POP-server into it. There’s plenty of discussion and instructions around the web so I won’t copy’n’paste them here, search for ‘fetchmail dovecot’ or something similar.