That's a very good point Phil. The resources are not always available. We do some of our validation currently in the production environment periodically. There have definitley been times where procedures have failed, but I would rather have a failure under controlled circumstances, correct the deficiency and succeed next time than to experience the same failure when I don't have the option to failback to a primary device.
To be honest though, I've run into restore process failures more often with physical servers than with any network devices, VMs or DBs. Routers and firewalls are just too easy to reload via script/ backup config which is where tools like Cat Tools and NCM come into play, but as byrona stated, the tool is only 20% of your management solution, the rest is process and procedures.