Transcript

Nenad Corbic: Good afternoon. I guess I’ll start. I think everybody’s still eating sandwiches. My name is Nenad Corbic, and I’m a chief software engineer at Sangoma. So, today, we’ll be talking about extending open-source PBXs for scalable media gateways. That’s a mouthful.

The topic is going to present the low levels of Asterisk. It’s quite different than the presentations we have today; it’s actually going to dive into the code a little and we’re going to look at some architectures of what is Asterisk made of from the ZAP-dial level and the TDM level.

We’re going to actually explore some limitations and the steps that we have taken to try to optimize and improve call performance, improve quality.

Pushing from Asterisk into the distributive level, I’ll also introduce the new channel in Asterisk, called Woomera that allows one to distribute Asterisk and really push Asterisk into a completely different area, where it hasn’t been meant to go.

So we start with the current limitations. Asterisk is one, big monolithic server. There’s nothing absolutely wrong with that. As Mark said, it’s pragmatic. And it’s a single-server architecture; therefore it’s limited by how many calls you can push through a single server.

TDM is known to be the biggest bottleneck of the VoIP-to-TDM architecture. SIP calls are lighter than TDM calls, just because TDM relies on hardware and you have to pull data out of the T1E1 line and channelize it and push it from the kernel space into the user space.

Software configuration has been a problem. It’s been solved now, through the hard work cancellers. DTMF and software HTLC, which used to be an issue, is now also solved. Software codecs is another big problem of a SIP-to-TDM gateway, and we’ll look at that as well.

Another one – not too many people talk about it – is the kernel-context penalty. That is something that’s quite important, especially when you’re trying to push through a barrier of over 250 calls on a single system. The kernel-context penalty really comes into play, and we’ll look at that as well.

TDM clustering solutions: Limited or hard to configure. It gets pretty complicated in configuration files, trying to cluster multiple Asterisk boxes; and we think we have a pretty good way, a pretty easy way, of clustering Asterisk boxes now.

Why are we doing this? What’s the whole point of this talk now? Well, customers, right? We all have customers, and they always push us, and they push us in more and more and bigger and bigger systems.

They see this open-source technology coming in, and you say “Oh! Well, I can show you how to do two T1E1 lines, 60 calls in one system”, and they say “Well, can you do more?” So that’s where we have to reengineer things a little.

So the current Asterisk PBD model looks something like this. I call these my “stick diagrams”. And everybody’s familiar with this picture, where Asterisk has multiple channels and each channel does its own thing; so one is SIP, one is ZAP. And Zaptel dial is the one that we’re actually going to be looking at here. And Asterisk lives on top of a Zaptel kernel module.

So the Zaptel kernel module really does the heavy lifting of talking to the hardware and channelizing all the voice channels and providing Asterisk with the devices, which Asterisk then grabs the voice media from.

If we open it up a little more, into something like this, it becomes a big mess. What is this? As you can see here, hardware lives at the bottom, and these are the TDM drivers – Sangoma, Digium, whichever.

And currently the Zaptel architecture requires us to provide one-millisecond chunks to the Zaptel kernel driver. Now, “one-millisecond chunk” means that I have to send eight bytes of data, from the hardware, into the Zaptel, for all channels. So what that really translates to is about 1, 000 interrupts per second per span.

Now, back in the good old days, when Asterisk just started, we had software echo cancellation – and we still do. And that is the reason why: the software echo cancellation really needs that one-millisecond timer in order to provide proper software echo cancellation.

Now this, of course, puts quite a bit of overhead on the system, especially when you start pushing a single system into something like over 200 calls or AT1s. So probably number one is the chunk size: how much hardware is pushing into the Zaptel.

One other problem there used to be is the software HTLC channel. Zaptel 1.4 now has this fix in; they have an actual option now to specify HTLC in hardware.

And the third red line is the kernel-context penalty. And, as you can see, let’s say we try to push this system to P3 level, where we have something like 600 calls, 600 channels in a single system. That would equate to 600 devices that the kernel will have to work with to grab the voice from each and every TDM channel.

And that’s impossible for the current – maybe these quad double triple cores now coming out might be able to handle it.

And that’s the beauty of this whole system; that’s why we’re all here: because we recognize the power of the idea, the motherboards and the new chips that are coming out. And that’s why we’re trying to push everything into software as much as possible.

So, what are the optimizations that we have made in hardware to battle the problems that we are seeing?

Well, the first thing we’ve done is put the echo cancellation where it belongs, and that should be in hardware. So echo cancellation: we have chips like Tezic [?] , like TI. They provide pretty good echo cancellation in hardware. So this has solved a pretty big issue.

And then, as soon as we get rid of the software echo cancellation, it really frees up a lot of things. Zaptel is not limited anymore to these one-millisecond time chunks that software echo cancellation was putting a limit on us.

So the second thing that we have done, right away, is that we have increased the chunk size in Zaptel. There’s no more point of just idling, because what Asterisk does is it says “OK, Zaptel. Start reading on this channel; but give me 20 milliseconds’ worth of data, in chunks.”

So that means that Zaptel will take 20 interrupts to fill up 20 milliseconds’ worth of data and then give it to Asterisk, and it’s kind of idle.

So what we have done is our hardware is able to do 10-millisecond chunks and we were able to take down the number of interrupts per second from 1, 000 to 100. So, again, when you’re working with 8-port cards – whether you’re working with two 8-port cards or four 8-port cards in a single system – it pretty much makes a big, big difference.

The hardware HTCL for the B channel has also gone in the card as well. So, that way, Zaptel is really offloaded at this point, as you can see. Zaptel is doing now what it’s really meant to be doing: it’s channelizing data for Asterisk and pushing up voice into Asterisk.

At this point, we have done pretty, from the hardware point of view, as much as we could to improve the hardware performance, put the stuff that needs to be in hardware in hardware.

But, however, one big problem that still exists is the kernel-context penalty. So, as I said, the greater the number of kernel devices, the greater the context penalty. The system doesn’t scale over 500 individual channels. So it’s really an exponential curve.

When you start pushing and pushing calls through, they just fall off at a certain point: the system will not actually gradually increase.

So the solution would be to per-span kernel device. Now what that does is – saying something like that drastically, drastically changes the architecture.

So if we were going to try to push more calls through Asterisk, a next step would be to change the Zaptel architecture quite a bit. That is something that it can be done and if somebody would like to do that, it’s possible.

And, I guess, the stage two is also memory map, the memory so that it goes directly into the Asterisk. So what we have done, the picture would look something like this. So, let’s say on a T3 system, if we had a T3 system, and we wanted to push all T3 600 channels, we would have something like… It would only equate to 28 kernel devices.

So that’s much, much better now. And the kernel would have no problem handling that data. But, the point is, one would have to have that span channelization inside of chan – zap in Asterisk.

So this is a brief overview of where Asterisk was, in the beginning, and where we have taken it, so far, to this level today. We don’t have this component, but Asterisk is able to run now. Echo cancellation is not a problem anymore.

Interrupt load is not really a problem anymore. And we have customers that scale two eight-port cards in a system, which is 200 calls with Asterisk, without any problems.

And we like Asterisk a lot. It works very, very well. And as long as you use Asterisk for what it’s meant to do, it works very well.

And I don’t agree with people that come in and say, “Well, Asterisk doesn’t work really well.” Well, you don’t know how to use it then, right? You have to use it in the specs that it provides. You’ve got to use the right tools for the job.

And that’s what we have done. What we have, gone the next step, is to try to – not to ask Asterisk to do too much. The point is to use many Asterisk boxes to solve your job. Don’t try to ask a single Asterisk box to do all the work for you.

So we came up with, a few years ago… I was at a conference and Craig Southeren from Opal started talking about this protocol Woomera. And I had no idea what it was about. Woomera is really just another signaling abstraction. What I saw with it, one can actually decouple the TDM hardware from Asterisk.

Now that’s a concept, like right now, everything, Zaptel, is very tight on a single machine. You cannot really decouple the TDM hardware from Asterisk.

But, using the Woomera channel, one can actually move a single Asterisk box and split it in two, and have an Asterisk with SIP on one machine, and just the hardware, the part, the component that uses so much, the most, CPU on another machine.

And without any extra configuration. Like with zero configuration changes to move, to separate, Asterisk and TDM hardware into two machines. And that profoundly changes the architecture of Asterisk.

Well, actually, it really doesn’t change from the user point of view. It changes nothing, because he still has a dial plan. He still has a channel that he dials on. The only thing is he doesn’t know that that channel is not actually not on the same machine.

So, again, the reason why we’re doing this is a response to a business need. We had customers coming who had an SS7 solution. And the SS7 really changes the architecture of the TDM. If we have to do this for PRI, we’d probably never do it. Because PRI, you have 24 channels, or 31 channels, and then you have one D channel for every associated, for every span.

But, in SS7, you would have one D channel that controls hundreds, even a thousand, of single, voice, channels. And then, the architecture really of Zaptel kind of doesn’t work in that case. We cannot serve thousands of calls on the same machine.

So, what is Woomera protocol? Well, it’s a really a signaling protocol based on TCP/IP. The media travels over UDP. And this is what it looks like. So, as you can see from the previous diagram, we have actually removed chan – zap and we have stuck in this thing called chan -woomera. So it becomes really an abstracted API to anything, to any signaling protocol.

And it talks in its own language. I’m just going to back to the next slide just real quick. And it has: HELLO, CALL, HANGUP, LISTEN, ACCEPT, DTMF, BYE. It looks similar, right, to this thing called SIP that does the same thing, but it’s not. And there’s a very specific reason why we did this.

We were 100% aware that SIP looks like this, but this is much, much lighter and it’s way, way less complicated. We never, from day one, we said this is not a replacement for SIP and would never be. This is just an abstraction to a TDM or any other protocol.

So a simple negotiation of a Woomera would be: a client would connect to a server and say hello. The server would say hello, I’m a server this and this. I can support BRI. I can support PRI. I can support SS7. I can support SIP. I can support whatever protocol you want.

And, at that point, the client would say, based on the dial plan, the dial plan would say place a call from the Woomera client into the server. And the server would magically route that call to whatever protocol was necessary and send back the client a UDP stream of voice.

So that’s all it cares about. Just give me the voice stream. I don’t care how you make the call. I really don’t care about what protocol you’re using. I just want the stream of voice.

So this kind of architecture actually presents… What I really liked about this architecture is the object oriented model. At this point, I can make my server super stable, small, compact, debug it once. And then I can also do the same thing for the Asterisk Channel Module. And once I debug once, I don’t have to…

Let’s say I wanted to go ahead and update, grab a new card, or let’s say I want to add a BRI card, for example. Well, I don’t change a single line of code in Asterisk, because that part has already been debugged. If I wanted to make a Woomera just an API, for a customer to run an API on, I can give a properly documented API. And I kill two birds with one stone there.

So let’s dive in a little more and see what it looks like. So, as I talked about, we’re actually in the process of writing an RFC for the Woomera protocol. It’s a joint project.

It’s a project from Opal’s side, Craig Southeren, from Opal, has written a Woomera interface for all of his SIP, H323, T38 stacks. And I have written the Woomera server for our TDM hardware for the SS7, BRI, and PRI.

So, moving on, this is our SS7 architecture. It just gives you one possibility of how to use this Woomera interface. So if you think about them, we have our TDM cards on the bottom, standard B channels and D channels. And the voice goes through the media gateway.

In fact, let me do it this way so you can see my… So the voice stream would pass through the TDM API. Now, one can use Zaptel here if they wanted. We have our own Zaptel-like kernel module that does channelization of all channels.

And it’s pretty cool, because you can actually open up a channel and say I want 20 milliseconds. Or open up a channel and say I want 10 milliseconds or I want 30 milliseconds. How much media you get is actually configurable per channel.

And other cool things like: give me hardware DTMF, or don’t give me hardware DTMF, or do tone detection. So you get to use all these cool things in hardware using this API.

So the media gateway, which is this piece, really acts as like a Woomera server on one end and then a signal translator on the other. And on the bottom here is the media engine that actually grabs the media from the hardware and then passes it to Asterisk.

And, on this side, actually on the SS7 side, we have our own SS7 stack that we’ve implemented. And this is all production-level code that works quite well.

So let’s just maybe take a call through and see how the call works with Asterisk. In the end, Asterisk is still the brains; nothing has changed. All this code here is really just a channel to Asterisk.

In Asterisk’s dial plan, you still do dial – now, instead of “zap”, you say “dial” – Woomera – group number, one, two, three, four, five, and the call gets placed. So, from the user’s perspective, really, they don’t see any of this, which is good for them; they don’t want to see it.

So an incoming call comes in – on the B channel, of course. At this point, signaling is signaling: here’s a call, on this band and this “chan”; you open up the media and do whatever you want with it.

So, in this case, in SS7 mode, an incoming call comes up – the red line, here – goes into the SS7 stack, and the SS7 stack tells me “Hey, you have an incoming call and it’s band 1, ‘chan’ 1.” At this point, the call goes through the Woomera and gets passed to Asterisk; and Asterisks’ dial plan says “Oh, incoming call! What do you want to do with it?”

Well, it says “I want to pass it to SIP”, and the call goes through SIP on the Internet, and you get an invite back and say “OK.” The call is placed, and Asterisk says the same thing: “Answer the call.”

And, at this point, the media gateway grabs the voice channel and sends it over UDP to Asterisk.

And, at this point, Asterisk has its voice stream, and you start doing whatever you want with it: you pass it through to SIP – or, if you didn’t want to pass it to SIP and you just wanted TDM to TDM, Asterisk turns around and makes the call back to here, into SS7, and braces it to a bridge, two legs. And you can make a TDM-to-TDM call here.

Now, all of a sudden, we have actually separated – at this point, this is IP. Even at this point, this is also IP, because the whole point of SS7 is that you can have one SS7 server handling thousands of media. So it would be wrong for me to literally bind an SS7 stack to the media gateway; it just doesn’t make sense.

So, as you can see, since all these are not hard-linked modules, one can now put this in one server; it could put this in one server; and then you could have another server right here, for signaling.

So we were actually able to split TDM from Asterisk and have been able to actually push it.

It opens up a whole bunch of new ideas because, again, object-oriented telephony – that’s what I call it – is that I can have all objects debugged once, test it once, and the stability of Asterisk drastically improves as the new hardware comes online.

If I want to add a new stack here, this piece of code has never changed. And even better is that I cannot crash Asterisk. I cannot crash due to my mistake. You can’t, because this right here is a socket.

So it’s one of those things: if I’m designing an Asterisk system and I want it to be a media gateway, and I want the channels to be super-debugged, tested – and usually the problem is just like in Windows. What crashes Windows is the driver, not the actual core. That can be debated.

Object-oriented telephony: So each piece is well defined; reuse of debugged modules. At Sandgoma, in particular, we’re a small company and we don’t have a hundred developers, and we have to be fanatically object-oriented in the sense that we do one thing once and that’s it.

So it reduces bugs; increases stability; and increases Asterisk’s stability, which is very important to us, because our customers are running 500-call systems, on a single machine, without any issues. Weeks with no rebooting Asterisk. So anybody who tells you that Asterisk doesn’t work for high load: it’s not true.

So, what does it look like in prettier diagrams here? This is our architecture. TDM is green, and stacks are on the red side. The server is the gray.

So, at this point, Asterisk talks Woomera to us, TDM; and also Asterisk can talk Woomera on the other side, which is Craig, for example. Craig Sutherland, Opal, SIP stack at H323. So this is a solution, for example, for Asterisk H323. If Asterisk really wants a stable, debugged H323 solution, one can use Asterisk with the Woomera to talk H238. This has nothing to do with the TDM anymore.

And Craig and I have been able to develop in parallel and without talking to each other. We just standardized on what protocol we’re going to talk to Asterisk.

So this is, for example, an SS7 clustered solution, where an SS7 box sits in the middle, and it places calls. A hundred calls go here, a hundred calls go here, and then all the calls can go to Asterisk. So this is the solution that we actually have right now, which we’re very excited about.

Another solution is Asterisk talking to multiple VoIP protocols, aside from the TDM. So, again, the possibilities are endless. We can attach any stack out there to Asterisk, if we just wrote this little piece of code. And you only have to write it once.

So all those old open VoIP stacks that are sitting there, while somebody doesn’t know how to write Asterisk or can’t go into the channel driver – because it’s quite complicated to do it – you can write your own little server very, very easily and attach to Asterisk.

Of course, all this is all open-source.

Now, this is a really cool one: the auto Asterisk load balancing and scaling. The native property of the Woomera server is that it’s a server, really. So you can actually have multiple clients connected and say “Hi!” Asterisk comes to a server and says “Hi. I’m a client. I’m ready to accept calls.” And say “OK”; so we start passing him calls.

And then another Asterisk stack can connect to the server and says “I’m also a client. I want to connect, but also get calls.” And Woomera automatically load-balances calls to each Asterisk box. Now, the really cool thing about it is that, if one Asterisk box goes down, you have the other one still running.

Now, the configuration to do all this is less than one line, because you just tell Asterisk where the server is. If you’re running this all on the same machine, you tell this guy that this IP address is on local.9001.

If you want to make this a dual-server architecture, you tell this IP address – it’s 19268, whatever it is – where the server is. And you just point them to this thing, and it just runs.

Now, this monstrosity here. I guess the Holy Grail is “Don’t drop a call”; right? A single Asterisk box can actually talk to two Woomera servers at the same time without any issues, as well as two Asterisk boxes can talk to a single server.

So, at this point, the calls in SS7, of course, can be clustered together so, if one SS7 goes down, the other one keeps going. So, at this point, everything is backed up; everything has a backup of its own.

And the calls can go up through here, up to one Asterisk box, or through here, to another Asterisk box. So, if any of the pieces go down, there is a backup for it.

So look what we have done. We have not asked Asterisk to do 500 calls. We have asked a single Asterisk box to do 1, 000 calls. We have asked one Asterisk box “Please, do up to 200 calls.” And that’s why I have a confidence in the system: because I know Asterisk works very, very well for that number of calls.

And, of course, on the SIP side, it’s all open. You can open the server or – we know all the techniques of clustering SIP. So that’s the whole point of what we are trying to say: use Asterisk for what it’s good at, and it will work well for you.

Where are we? I have maybe ten more minutes left. We have some pretty cool ideas here, also. We have a little RTP TAP; I was going to maybe talk about it if we have time for asterisk voice recording. And what it does, it can pull all the channels of voice on the side and send to adjacent Ethernet card in real time. So, one can actually do voice recording asterisk.

8 E1’s with four percent system load because everything is pushed internal to another Ethernet device. There’s an actual company making a product out of it right now.

I want to also reiterate that gate made a good point that there is a switch. The reason… now, the whole point of this architecture also implements the use of the solve the kernel context penalty, because the Woomera server talks span based to the hardware.

So what that means is that one can now run on a single server many more devices. E8 has proven this because E8 does that. E8 runs span based API to the drivers and they are pushed over 30 E1’s in a single machine. I think they hold the records still. So, maybe we’re going to have to…

Anyway, so, this is the architecture and I think I’m in time, distributor performance. What is the performance of this thing? Well, I guess this is old. I mean it has received 566 calls so our point is that Woomera lighter and smaller again it is not a simple replacement.

It’s only meant to be run on a local network. It’s meant to distribute asterisk on a local network. If you want to talk Internet, use zip, you want to use G729. It’s only meant to be on a local gigabit LAN.

Demand is growing with customers all over the world asking us to do more and more and more calls. And they want to use open source as a talk upgrade equipment and that is what we are trying to give them. Thank you. Any questions? Comments?

Man 1: Are you giving away fifty cards?

Nenad: I don’t know. Doug is down there, you should talk to him about that.

Man 1: I will. [laughter]

Nenad: So, this might be even too much.

Man 2: OK. I see that you are mentioning something called Sangoma here all the time, but I haven’t seen that inside from the source distribution. What is the status here?

Nenad: I am the maintainer so I don’t have any problems putting it in a… If you say, yes, I would like to have it. I would be more than happy to plug it in there. It just makes my life easier.

Man 2: I am one of the developers that isn’t employed by Digium so personally I would like to see everything that connects to asterisk and can be maintained properly and a company that says that they want to continue maintaining it. They want to see it in the asterisk.

Nenad: Yes and the beauty of is this channel I’ve had it since 1.2 and literally there was three line changes for 1.4 and it just ran, so, this channel is quite…

Man 2: That stuff is what we like to hear. Randy, did you get the recording that upgrade to 1.4 wasn’t as hard as people say… And all the change is good?

Man 3: Not that I knew it.

Man 2: No, no, no. All the changes you made to Chan Sap are they being contributed as well? That fan based?

Nenad: No, we haven’t done that. That’s the thing; we haven’t done that in Chan Sap.

Man 2: So you are showing promise work?

Nenad: Well, that is an option for somebody if somebody would want to do it in Chan Sap. That is the way to do it, but we already have APIs for that so it’s… But Sangoma has already those APIs, so we already have that, so we have to service our customers right away, so that’s kind of…

Man 2: I look forward to seeing those changes too.

Nenad: OK. Yeah, all our stuff is open source so do it anyway.

Man 4: As is ours.

Man 5: Is that some sort of a do all issue? This stuff is all GPL, isn’t it?

Nenad: Yeah, all the stuff is GPL.

Man 5: Could it become part of the Digium anyway because it is GPL?

Nenad: Chan Woomera is completely GPL and should be part of asterisk.

Man 5: Yeah, because it’s Woomera’s, we don’t have possibility.

Nenad: Say it one more time.

Man 5: You do own the source codes?

Nenad: I do… Oh, yes… No, well actually who owns the source codes? Tony Manasali wrote part of it. But he won’t have a problem. I wrote most of it and I got to production.

Man 6: I have a question, the ad work said no one would be willing to sell it, So it could be rolled into converted to a license where Digium and put it in an asterisk business edition and sell it? That is the real crux of the question.

Nenad: Again, in the end I don’t have the problem with asterisk is using it because this channel is really useful and it is open source so… Selling it and all that stuff. I mean that’s fine. That is a different issue but the point is that customers… They should be part of asterisk and use, if somebody wants to do it. The cool way that this channel will give you is the ability for somebody to write a simple channel driver for themselves, so they don’t have to go and rip asterisk apart, put their channel in there, and break asterisk.

Man 7: So [off-mike speech] .

Nenad: It’s a zip driver too. It connects to a zip stock as well as in a previous line up here.

Man 7: Yeah but it is limited in regards to do all those updates in the channel drivers than just [inaudible 35:55] cirlces.

Nenad: Absolutely. For example, our BRI right solution for this, right, it is same thing. And we do a lot of telephony as well. This channel makes music and all. It does all the PBXC stuff and it’s pretty. It’s all there. So, cool thing is there are BRI solutions literally; there was zero coaching to asterisk when we roll out the BRI card. Because there was nothing changed on the asterisk side and there was actually zero change here as well. So, we throw in a stack and a reuse of code… Anybody else?

Man 8: Does it work in asterisk or can I..?

Nenad: Well, anything you write to Chan Woomera for a client for can use this whole architecture and our … We have a lot of customers that want to use API, for example, and this is one way of providing our customer a global API that will rival Dialogic. It’s, place call right? I don’t care what it is. Is it seven digit code or whatever? It’s just place call and did you get a UDP stream of voice.
That is what the holy grail of dialogic is, right? So, that’s kind of…

Man 2: Can I ask one last question? Many of our clients are using BRI and analogue internally, before using, because In Germany many people who are still using analogue faxes. And PSD an is BRI ISD an. And you have some sort of special thing about that. What is it and what makes it so special?

Nenad: Our BRI?

Man 9: No, the whole faxing thing. On your website, it says that we guarantee fax and ditching and … Why.

Nenad: We have a hard solution for fax word, we can synchronize the clock from a BRI card to an analog card or from a PI card or E1 card to a BRI card or from and E1 card to an analogue card. So we could push the clock from card to card, right and that is the secret of faxing. I mean it’s not rocket science. But you need to synchronize the clock on two cards in order to send the fax reliably, otherwise you get bid airs. So that is one of those things that we try to improve. That is how we try to improve asterisk.
[silence]

Anymore questions? So, going back to your final answer, I know, I am overtime. And yes, you can if you so… We plan to export channel Woomera to other open sources PBXs. I don’t see why not. So, if they want to do it. We will support everybody. We play nice with everybody. Thank you very much for your time. [applause]

Yes and that was Windows, I know. I forgot to take it out first.