This looks like the action had nothing to remove and just ran without touching the files. If you use extended tags, the tag of each file is always written.
So when comparing performance, please make sure that all files are tagged with an album artist tag. I agree that there is a difference in performance, but not to this extent.