Returning Literal '&' Character CURL Realtime

[chance]

Am pretty new to Asterisk but a fairly seasoned programmer. Am trying to figure out how one would be able to return a literal ‘&’ character using Asterisk’s real time architecture. While there are many ways to accomplish dynamic configs with Asterisk (AMI, AGI, CURL, ODBC, etc), I am currently relying on using ODBC realtime for the core configs, and using CURL for dialplan-related configs. Each of these methods are successfully returning data in the line-by-line method like:

category=something&cat_metric=something&commented=something&var_metric=something&var_name=something&var_val=something&filename=something

This method is okay and we have realtime working very well. But of course there is always one little thing…I would like to return data to Asterisk using the working ODBC and CURL methods but am finding that if I return data via CURL that contains an & character in the var_value data, Asterisk does not like this. Reviewing the res_config_curl.c script shows me that Asterisk uses the & character to exclusively determine the field pairs returned from realtime as outlined in the category= line above. So an & in the var_value data just makes asterisk think it has more fields than it really does.

Two instances where realtime data return with &'s in the data is failing me:

1. I am defining hints for SIP extensions. It is very easy to do, but I want to include a standard extension state hint (ie: SIP/nnn) AND also a custom hint for DND (ie: Custom:DNDnnn). In flat file, this hint would be combined as:

exten => nnn,hint,SIP/nnn&Custom:DNDnnn

But when we issue dialplan reload we end up with:

exten => nnn,hint,SIP/nnn

This produces NO error, but we do NOT have our custom DND hint defined as desired.

2. I am defining an outbound context that handles invalid dialing patterns. I want this special context to playback a series of audio files if an invalid pattern is matched. The reason the context is dynamic and not programmed in flat extensions.conf is because we want our PBX to prevent users from altering allowed dial patterns. So this [outgoing-restricted] context is loaded dynamically and always contains the invalid patterns. When invalid dialing pattern is matched, we want to run Playback() application using combined audio files. How does Asterisk’s Playback() application expect multiple files? Via an & character, so the realtime return should look like:

exten => _011.,1,Playback(im-sorry&but&the-number-you-dialed&is-curntly-unavail)

But when we issue dialplan reload we end up with:

exten => _011.,1,Playback(im-sorry

And Asterisk produces errors due to missing trailing ).

It seems when attempting to return these values using CURL (which uses the & character to delineate fields as outlined above, the & characters are being viewed by the realtime parser as field separators and not part of the data value, which of course truncates some of the returned data.

While we can easily return one line for every audio file in Playback(), it is even more cumbersome than already, and same method will not work with hints because there can only be one hint (-1) priority for any one extension.

Does anyone have an idea they can share with me to help me overcome this? I know that Asterisk can simply be recompiled and I could change the separator character in res_config_curl.c from an & to something else but I am hesitant to do that and am hoping someone else has run into this in years past and knows an easy work around.

Thanks in advance…

[InterLinked]

Maybe encode the whole thing and then decode it into Playback?

I haven’t done that with Playback before, though I have a few subroutine lookups for playback.

Don’t forget that Playback needs the ,noanswer option for intercept messages.

[chance]

Thank you for this idea. I had already tried using the various encoding functions like htmlspecialchars() and url_encode(). They did not seem to help because in the end, they generate the same & character in the readable output. So Asterisk has no way to discern an & as a field separator vs. part of a field’s data.

The issue I have is that Asterisk does not have a mechanism I can find on how to change the field separator character that Asterisk uses to break apart field/value pairs (ie: field1=value1&field2=value2, etc.).

If we could change the separator from an & to say a ^, then I could return an & in the data because Asterisk would then use the ^ to break apart the field/value pairs (ie: field1=value1^field2=value2, etc.) and not the & character.

[chance]

I just reread your idea…when you say encode the whole thing, are you referring to encoding the multiple audio files into one audio file, and then passing that single audio file to Playback()? If that is the case, that is a good idea–I could just use something like Audacity to merge the audio files together into one. And that would be a solution for the Playback().

Although that would not work for the case where I need to combine several hints together where the hints have one or more & characters in them.

[InterLinked]

Sorry, no, I meant more the first thing you said, encoding the characters. I wouldn’t combine files since that limits flexibility greatly.

Another thing to consider is that often curling or doing shell calls will introduce junk characters like new lines which you can get rid of using the FILTER function.

Since I can return a value containing &, it does not seem there is a fundamental limitation on doing what you are trying to here.

[chance]

Are you saying you CAN return a literal & character via asterisk realtime where asterisk expects the return to use & to separate field/valie pairs?

If so, what script engine are you using to do so? PHP, Python, etc? Would you mind sharing the code logic that outputs the result expected by asterisk that also includes an & as part of a data value?

[InterLinked]

I have never used Asterisk Realtime. Typically, if I am retrieiving data from a remote system it is using curl, e.g.

same => n,Set(var=${SHELL(curl Example Domain)})

Often, I FILTER() the shell output, especially if it includes line breaks or carriage returns.

Then, with that in a variable, I can return that, put it in an array, cut it, or do whatever I want with the data.

As for why I don’t use the CURL Asterisk function, that didn’t work when I started using Asterisk but SHELL(curl) did, and I haven’t tried the CURL function since (though I probably should!)

[chance]

SOLVED

The issue we faced was that we are not using CURL to return a single value in the dialplan, so using the CURL() function was never part of the problem.

We are using CURL with realtime external configurations. The calls to our PHP script are called by Asterisk via extconfig.conf, and an Asterisk parsing function named *config_curl(…) within res_config_curl.c. That function invokes the URI to the server, and the server is expected to return multiple config lines. Each line is supposed to return something like this:

category=hints&cat_metric=0&var_metric=1&var_name=exten&var_val=101,hint,SIP/101&id=1&filename=extensions.conf&commented=0

The ‘&’ character is treated by Asterisk as a delimiter separating key/value pairs. The above shows that we are defining an extension hint for extension 101. But only ONE hint can be defined using this approach. What if there were more than one hints for a single extension? Asterisk expects multiple hint defines to be separated by ‘&’. If we want normal extension state, AND a custom DND state in our hint, Asterisk would expect this to be defined as follows:

exten => 101,hint,SIP/101&Custom:DND101

To define the above in CURL realtime, we have to return a literal ‘&’ in our var_val pair. But Asterisk’s realtime CURL parser treats that literal & between SIP/101 and Custom:DND101 in the data as a key/value separator. And we end up with just:

exten => 101,hint,SIP/101

The second part of the hint &Custom:DND101 is lost in parsing.

We solved this problem by editing the source res_config_curl.c file. Specifically the *config_curl function. And then recompiled Asterisk with a simple ‘make’ command. For Asterisk 16.16.2, the line edited originally read:

while ((pair = strsep(&line, &))) {

We changed this line to read:

while ((pair = strsep(&line, [&]))) {

Notice that we have told Asterisk to use ‘[&]’, rather than just ‘&’ as the key/value pair separator. Then in our PHP script that returns the realtime CURL config, we return this instead:

category=hints[&]cat_metric=0[&]var_metric=1[&]var_name=exten[&]var_val=101,hint,SIP/101&Custom:DND101[&]id=[&]&filename=extensions.conf[&]commented=0

Now when the dialplan is reloaded, and Asterisk requests the realtime CURL config, our script returns data using [&] as the key/value pair separator and Asterisk properly parses it. And when the var_val requires a literal & in the data value, as we do with multiple hints, Asterisk leaves them alone and treats them as part of the value.

The only downfall to this approach is that if we ever upgrade Asterisk, we have to remember to modify the *config_curl function within the newer version res_config_curl.c file.

Hope our solution helps someone else…