Every user in each conferences can hear or speak together.
Can a user be in many conferences?
Or can i customize the rules for an user ? (for example : User 1 can speak and listen to User 2 but User 1 can not speak and listen to User 3 even if they are all in the same conference)
I imagine it will be difficult to find phones with sufficiently good echo cancellation to avoid cross talk between conferences, even if you could configure it.
To the extent that you can solve the first part by echo cancellation or VOX, someone in {Red} might speak on top of someone in {Yellow}, as far as someone in {Red}∪{Yellow} was concerned, without being aware they were doing so (hidden talker problem in CSMA radio networks).