Unit tests cover the single functionality but ignore the system as a whole. Most regressions I've seen in industry are because of a lack of understanding how the system components interact with one another.
Therefore, I see unit tests as one pillar but also suspect that without good quality integration or end-to-end testing you won't be able to realize the riskier re-factors you describe. Perhaps you consider these part of your regression testing and if so, I agree.
The way I like to view it is that Lego bricks might pass unit tests aka QC with zero issue, but you can still easily build an unreliable mess with them.
Therefore, I see unit tests as one pillar but also suspect that without good quality integration or end-to-end testing you won't be able to realize the riskier re-factors you describe. Perhaps you consider these part of your regression testing and if so, I agree.