I have Asterisk speaking to a Prologix/CM v2.1 Avaya platform using an H.323 trunk. Currently I am using ooh323 which comes in the ‘asterisk-addons’ package. One issue with ooh323 is that in only supports rfc2833 (out-of-band) DTMF handling, while the Avaya either likes in-band (over G711) or some proprietary out-of-band protocol. Therefore DTMF passing between the two is an issue. I have found that if you use chan_woomera this is not an issue as it supports in-band DTMF handling.
I currently have the Avaya and Asterisk connected using an H.323 trunk and it works fine. There are issues trying to use the Avaya as a gatekeeper (to be able to connect Asterisk to stations instead of a trunk), more details on the Avaya as Gatekeeper issue here.
Now, I do not know much about the Avay IP Small Office solution, and it may be entirely different. But you may go to the aucommunity.com website to find out more specifically.