Sounds like it would be very important to know the "verbiage of the original contract." Why is this not available to you?
All of the work-for-hire I do with major companies explicitly state that any source files I create are their property. That is when I sign a contract. If it was a verbal agreement or a deal hashed out over emails, unless stated at the beginning, the source files belong to the freelancer as their intellectual property. How you proceed obviously depends on what this guy signed. If you can't find that out, and it will make your life easier, pay him. It's not worth the headache of threatening and intimidation. Which never works anyway.
-TG