Keeping or removing the legacy specular layer makes no real difference to render times, when you add a Beckman/ggx layer, these still incorporate specular as part of the reflection.
Specular does have uses, but if you are going to the lengths of adding real reflections and using roughness to soften them, at this point imho you may as well go all the way and dump specular completely. Spec is just a fake reflection of a light source, you may as well just add a real reflection and make a real object for the reflection to use. It certainly doesn't justify the render times.
re: turn off color and just use a diffuse reflection; I can't agree here. This increases render times dramatically. Unfortunately neither physical nor standard render are that well geared to doing diffuse lighting with reflectance. I've done quite a few tests and generally sticking with the older color channel + gi lowers render times by an order of magnitude. For the scene which prompted this video, I get render times of 20 minutes with GI but no specular. 1 hour with GI and specular, but then if I switch to a reflectance based diffuse system my render times exceed 10 hours even with just 2 reflection bounces. 3+ reflection bounces gives absurd render times.