I found a way to do it on the Wiki tricks+tips page.
I modded this for my case:
[dialout]
exten => s,1,Answer ; Answer the line
exten => s,2,Wait,1 ; Wait a second, just for fun
exten => s,3,DigitTimeout,5 ; Set Digit Timeout to 5 seconds
exten => s,4,ResponseTimeout,15 ; Set Response Timeout to 15 seconds
exten => s,5,Read(tfnumber,what-number)
exten => s,6,dial(SIP/${tfnumber}@sip.broadvoice.com,30)
So, whenever I dial in from the phone on the pots line I check for the caller id number, and if it matches I goto dialout context.