Delay in silence detection while using TALK_DETECT

I am using asterisk 15.5 along with official python library for ARI.

I am using function TALK_DETECT for detecting speech or silence on the channel. In ARI, I am using ChannelStartedTalking and ChannelStoppedTalking to detect speech and silence on the channel respectively.
While using default threshold value I am able to detect speech almost instantly and accurately but It takes some time for silence to be detected.
I tried to meddle with the threshold value, even when I try to reduce it to 1000ms I have noticed that It still taking approximately 2 seconds to detect silence.

Is there no way to make detecting silence quicker?