What's your motherboard and cpu? Are you running AE 11.0.2? I have had similar issues using an older mobo and the newer nvidia drivers with a 680. Rolling back to 306.xx did solve the problem. You could also try: deleting your AE prefs, reinstalling AE, uninstalling GeForce Experience or trying the nvidia driver cleaner + reinstall clean 320.xx
Just curious, why wouldn't you use Premiere Pro CS6 instead of Vegas if you have already production premium? It's much more powerful.