Its trickier to scale for sure. I have actually seen something very similar work really well at an early startup. The team was around 5-6 people and fully remote at the time.

The real key is communication and trust. It only worked well for us because we communicated frequently, the CTO/lead was really good at acting as a conduit between everyone working on related projects, and we all really offered each other the trust to let each other run with it and adjust on the fly.

It was really common for us to go into a week with a clear idea of what the next feature to build was, only to end the week with the person working on it finding a different path or a reason not to build that at all, often with an alternative to propose. We'd discuss it as a team when needed, but it was always clear that the person who was working on it was the expert on it at the moment and had a lot of sway in direction for that feature.

It isn't easy, and it did actually fall apart when the team scaled to 10+ people, but there may have been ways we could have adjusted and avoided that too.