You make an excellent point but as it turns out the CMB would still appear uniform even the speed of light was not the same in all directions. The reason is that if the speed of light is different in one direction, then the effect of time dilation will also be different depending on direction as well and these two factors cancel one another out resulting in what will still appear as a uniform CMB.
The below paper goes into complete technical details giving an example of the speed of light being c / 2 in one direction and instantaneous in another direction (so that it averages out to c), and how the differing time dilations result an isotropic view of the universe.