Voicemail- Pollmailboxes- MWI Issue

I am having an issue with Asterisk voicemail MWI notifications being inconsistent. I am using Asterisk Voicemail with IMAP and pollmailboxes=yes. I am trying to achieve “Unified Messaging” in this way. My issue is that when a user is listening to a new message through the phone, the MWI light turns off due to pollmailboxe settings, but then once they hangup (without saving or deleting) the Voicemail dialplan application turns the MWI back on.

Is there anyway to only use pollmailboxes events to send NOTIFY and not other app_voicemail driven events?