Multiple login from same phone asterisk

I am having a problem with asterisk where after a user joins a conference room press the hang up button (not hold it) and then dails back in to the conference room from the same phone asterisk seem to join both users in.

when i checked it from the cli i could see 2 users with the same caller id joined in to the same room.

So the question
How do i stop that from happening? Basicaly I would like to kick the first one out and login the second one.

Can anyone please help me on this regard, i appreciate your help.