I would guess that the stacked booles and the various modes chosen therein, in combination with an incredibly small voxel size in the the VB are what is breaking that. I remain confused about why there is any boolean action needed here at all when using the volume builder, which viably replaces all of its functionality !
Here is what happens if you simply put the Cube, and Right and Left 'Big Circles' directly under the VB with no booles required... now your rotation segments have an actual effect on the smoothness....
After that, you can start to subtract other operands, also children of the VB, which is how I removed the small circle block on the left there, and could go on if so minded to complete the whole model using this sort of approach, which is how the Volume Builder workflow is meant to work...
To get the result above my stack is simply this...
Now, as you have seen, booleans can be used inside VBs but I have not yet found any circumstance where doing that helps the workflow more than confounding / obstructing it, or making it needlessly complex and even more calculation-full !
We could also talk about whether VB is the best approach for this kind of object - and I would say it is, IF the primary focus for the project is massive speed at the expense of polygon efficiency !
If you had a bit more time to throw at it you could model this via regular Sub-D modelling (no booles) quite easily and you'll get a nicer overall result, with about a 1/100th to 1/1000th of the polygons required to do it the VB way, and it would also be eminently more suitable for easy UVs and any subsequent animation, were that to be required... just a thought...
Hope that helps
CBR