http://www.trixbox.org/forums/trixbox-forums/open-discussion/daily-reboot-script
chris
Posts: 209
Member Since:
2006-05-31
Suggestions for deadlocks
Wed, 12/12/2007 - 11:14am
new
Deadlocks are an absolute scourge, so I feel for you! They plagued Fonality pretty heavily from 2004 to 2005 in our larger installations (call center or not). We finally got them almost completely solved by using a combination of approaches:
1. Threw out Asterisk's chan_agent and rewrote an new agent system from scratch.
2. Rewrote about 10,000 lines of core Asterisk code -- code which reliably produced deadlock scenarios in our hammer/anvil. We call our modified/improved/hardened version of Asterisk "PBXtra core".
3. Modified the AMI quite a bit (to produce XML amongst other things). This, BTW, is what HUDlite reads which is why it doesn't work as well in CE as it does in Pro.
4. Modified our GUI to disallow a number of other known deadlock scenarios that we could not directly fix in Asterisk.
5. Wrote extensive monitoring and self-healing code for "pre-deadlock" conditions within Asterisk, so that the system can recover on its own without a reboot.
With all of these modifications, we were able to host a single box doing 35,000 Queue calls per day. On our own single box at Fonality we do roughly 3,500 calls per day, many of which are queue calls.
Now, there *are* still some conditions that create deadlocks and each new version of the PBXtra core fixes more of these. So, I wouldn't declare we are "done" - but we have made a ton of progress over the years.
So, my advice to you is:
1. Use your idea of rebooting the box nightly - if that doesn't bug you. It's a cheap fix. ;)
2. Switch to Pro. Pro has a very strong call center solution - with more features (and bug fixes) than are found in the native chan_agent that we replaced including proper handling of concurrent queue calls, very strong call center reports, barge/monitor/record with the touch of a button from HUD Pro, etc. But, this will cost you $.
3. Pay for some Fonality trixbox CE support and see if our guys can't coach you a bit on either the deadlock issue or a clean reboot cronjob. I would ask specifically for Andi Chi when you call in.
I hope I have helped a bit. I spent a lot of my last couple of years (and got a lot of gray hair) working on these dreaded deadlock issues, so I thought I would tell you what I have learned in the process. ;)
--
Chris Lyman
Fonality CEO / Janitor
My new CEO / Janitor blog