At first I assumed this problem was related to just my build, but after doing some googling, the only other people having the issue, also have PowerEdge T110 IIs. I found this article on the Dell Forums about it. In my experience with this issue, the first time around I attempted to install the Hyper-V VM using the included Dell Windows media. I know this was against my better judgement but I was hoping to save some time by not having to download a fresh ISO, and I figured…how bad could it be?
The install took incredibly long, and afterwards seemed to hang on Finalizing your settings. I gave up after a while and downloaded a fresh RTM ISO of Windows 2012 and installed that. Although I ‘feel’ like it installed a bit faster, it still definitely hung on the Finalizing your settings step longer than usual. I can only theorise why however, perhaps some sort of interaction with proprietary hardware with virtualisation capability?
Regardless, the answer is just wait a long time. Once it installs the VM seem to operate responsively and quickly, just appears to be an installation quirk.