Read external files

Asterisk Asterisk Support
1

Hi all,
i need to implement a extension context where call comes in and check external file csv. If calling number match the one present on a certein column, i need to take 2 parameter: new called number to call and a new name to replace the actual calling number display name.
My question is, what is, as per today, the best way to achive this result? Is there a tool in asterisk to read external files? or i need to use tools like AGI/AMI ?
What do you advise?

Regards

2

AGI or SHELL. Shell needs safeties disabling. None of these will be CPU efficient.

3

Thank you,
since both are not CPU efficient, which method would be better to use to also have adequate performance?
Thank you

4

On Monday 15 April 2024 at 14:46:04, spady7 via Asterisk Community wrote:

Hi all,
i need to implement a extension context where call comes in and check
external file csv. If calling number match the one present on a certein
column, i need to take 2 parameter: new called number to call and a new
name to replace the actual calling number display name.

  1. Approximately how many records are there in your CSV file?

  2. How frequently do the records change?

If the answer to (1) is “no too many” and the answer to (2) is “not very
often” then you could consider loading the data into Asterisk’s internal
database and doing the lookup there.

Antony.

–
There’s a good theatrical performance about puns on in the West End. It’s a
play on words.

                                               Please reply to the list;
                                                     please *don't* CC me.
5

The difference in resource consumption should be similar.

The shell() function is easier, the agi() application is more featureful.

shell() will require parsing the returned values in dialplan.

agi() can set channel variables, avoiding parsing and quoting issues.

Writing the executable (for shell() or agi()) in a compiled language (eg, C) is xxx more efficient than one of the P languages. You probably don’t have the call volume for the executable execution time to be meaningful.

6

CSV file Will have thousands of records and it will be updated often. So I exclude using internal db.

@sedwards can you give me some more hints using external language to parse file and results?
You intend create a separate script (in C or Python) and then recall it by dialplan?
Am I wrong?

Regards

7

While I would classify myself as a c-weenie, I’d probably write this as a script first and then decide if the performance and number of executions per interval justify the effort to write in C as an AGI.

If your data has quotes and embedded spacing, you may want to look for a “jq’ish” command line tool to parse the CSV. If not, awk or grep+cut can probably do the trick. (Note grep has a --max-count option so you don’t have to process the entire file every time.)

I’ve never used the shell() function, but this seems appropriate, Note that the cut() function defaults to using ‘-’ as a delimiter.

However, it’s been my experience that ‘just do this one little thing’ can morph over time into mission critical subsystems, so you may want to plan ahead. Much more than 2 items being returned or incorporating more business logic and I’d probably start writing an AGI.

8

For a non Asterisk related project I am querying a 37.000 lines csv in a shell script with

cat filename.csv | grep -i searchterm

and then divide the result line with cut. While of course this is a very quick and dirty approach, it works surprisingly fast, even on a quite weak machine.
So I am sure this could work well if you do an AGI script that works like this. Still a database based approach would be better without question.

9

Thanks guys, i will have some tests and see how it will works.

10 
grep searchterm <filename.csv | cut --delimiter=, --fields=2,3

Eliminate cat. Since the OP is searching for a number, ignoring case has no benefit. You can add ‘–max-count=1’ and grep will stop after the first match which should cut your average search time in half. Assuming no funky stuff in the CSV, cut can be used for output.

awk --field-separator=, '/5551234567/ {print $2 "," $3; exit}' filename.csv

Eliminate cut. Again, assuming no funky stuff in the CSV, awk can do it all by itself. The ‘exit’ causes awk to exit after the first match.

11

I would always use SQL database in such cases. ODBC works fine with mysql for example.
For example: when we have an incoming call we check if we need to record this call

exten => s,n,Set(ARRAY(CALLERID(name),RECORD_CALLS_CH)=${ODBC_LOOKUP_NAME()})
...
same=>n,GoSub(subTurnOnRecording,s,1,(${RECORD_CALLS_CH}))
12

I tried with AGI and external python scripts and seams to work very fast, even csv file is aroud 20000 raws.
@RandomPBXUser your solution is always fast? You never get some delay in processing incoming calls?
Regards

13

The cost in using AGI is that of launching the application and its interpreter, and I think a shell, so most of the cost is up front, unless there is a very long file to read.

14

It depends on SQL query. If you have an index on the column you are searching it is fast. I do not have tens of millions rows in my DB.

15

If the text file is sorted, and the key is the first field, you can use the “look” command to find the the correct land in O(log(n)) time, for the actual search, as it uses a binary chop search.