E-mail support with humor

I am maintaining a BMC AR-System environment with ITSM applications supporting ITIL applications including Incident, Problem and Change. This system is sending out a lot of e-mail notifications to all the users. Quite often the end user reporting an incident answers the receipt e-mail asking questions or sending additional feedback. Very often the system just receives autoreplies about the users being on vacation or on paternity leave.

The autoreplies we don't care about but the questions and feedback can be interesting to pay attention to. I want to build some workflow to filter out the interesting stuff and attach to the relevant ticket in the system or automatically deal with them in some other way in the cases where it is not possible to find the corresponding ticket. So far I didn't get the permission to do this development.

Instead I was asked to set up an autoreply in the exchange account for every incoming e-mail that comes from within the company. In this autoreply we are just going to inform the end user that they've sent an e-mail to a system and no person will read the e-mail and that they should send the e-mail to some support address.

So I did, but I was a little bit concerned that Exchange was not smart enough to deal with the auto-replies from end users on vacation. If our system sends an e-mail to a user on vacation and and the users e-mail account sends an auto-reply to our system that send an auto-reply back to the user that sends an auto-reply again and so on... You get the picture. It will become like a ping pong match.

I sent an e-mail to our e-mail support group asking about the risk of an ping pong match. I really liked their answer.

"Usually Exchange is smart enough to not start a ping pong match that it can not win. But occasionally it can happen that they get excited and someone smashes in a backhand volley."

OK it didn't wipe away all of my concerns, but still I really like to see some humor at work. :)

New server on the way

My old Linux server died. First a fan stopped working so the computer froze now and then. After not so many days it refused to start at all again so something in the hardware is fried. I guess there is something on the motherboard that burned up. I really hope there is no real damage on the hard drive. That would be a small disaster because I have no fresh backup.... :P

So now I have ordered parts to build myself a new server. First time ever that I will get a new computer as server. Previous I have only taken some old computer that is ready to throw away and made it to my server. The one that just died was my fathers before I asked if I could take it when he bought a new one. He had even bought a new one after that switch. My dead server is probably over ten years old and deserves to rest in peace.

This is the specifications of my new server:

• Antec mATX Cube NSK1380 - (350w)
• Gigabyte Socket AM2+ mATX AMD 780G (GA-MA78GM-S2H)
• AMD Athlon 64 X2 5600+ 2.9GHz / 2x512kB Socket AM2 (65W)
• Corsair XMS2 Xtreme 2048MB DDR2 XMS2-6400 800MHz
• D-Link Nätverkskort DGE-530T 10/100/1000 (PCI)
• Samsung Spinpoint F1 1TB (7200RPM / 32MB / SATA II / NCQ)
• Samsung SATA DVD±RW 22x - Svart (Dual Layer) (SH-S223F)

and a

• D-Link 5-Ports Switch DGS-1005D 10/100/1000

I didn't do so hard investigations about the hardware so I really hope I didn't make any bad/stupid choices. As long as it works well I don't care much about the performance. My previous server had 350Mhz CPU and 384Mb of memory and that worked very well except in the graphical X Window interface using KDE which was a bit flickery and slow but still usable. Maybe that pain was mostly because of the really bad graphics card.

I am not sure yet if I will use the hyped up Ubuntu or the old reliable Slackware that I have always used before. I have actually used Slackware since summer 1994 and am quite comfortable with that distribution. I have used Ubuntu on my dual boot workstation and think there are both good and bad things about it. Most things I feel that is bad is probably my lack of knowledge about Ubuntu so I should probably choose that one for my server so I learn everything I need to know. But it feels almost like betrayal to not use Slackware any more. Hmm.. so hard decisions... What should I do?

2 hours of panic

Just after most of my colleagues left work for the day our production ARS server stopped working. It is impossible to access the system in any way. I check the error-log and find that the service crashed five times in a row. Then my connection to the machine totally freezes and I can not even read the log. Still no access to the AR-System.

Panic!

I call the server maintenance (SM) and ask if they notice the problem. They say they do and that they have informed the helpdesk. I explain what I experienced and ask if there are any planned changes that are supposed to get implemented this evening or any other known activities. No there were nothing. It was getting more scary and it started to point at there was something terrible wrong with our system. Good that Girlwing is at a course tonight so it doesn't matter if I work late...

The server suddenly comes alive again and the SM-guy logs in and tries to checks the server information. The server freezes again and nothing can be done. We keep talking about what can be the problem and I am start having awake nightmares about evil viruses.

When the system was responsive I searched for all the information I could find about imports and reports and nothing was supposed to run at this hour. So I was starting to rule out heavy load on the ARS service. I relax and the system got unresponsive again...

A couple of minutes later it is again possible to do simple searches in the ARS and soon the Windows OS is workable again. The CM-guy continue to check the server information. The CPU is running 100% and the memory is all eaten by a bunch of cscript processes. Then everything freezes again... it keep doing like this all the time while we are talking about the incident and searching for the cause.

Then suddenly after two hours of hard work he say... "The system hangs every time I relate those incidents with each others".

I am silent for a couple of seconds... then I say... "So if you stop relating incidents with each others, the system will work well and helpdesk will be able to access the incident application?".

He agrees to wait a couple of hours to continue and I rest my case...

...

The thing that had happened was that some computer had been switched off before the monitoring had been taken off and hundreds of incidents had been automatically generated and send to SM and filled their support group basket. There is a lot of work to manually close each incident because you have to fill in closure code and cause description. But instead it is possible to select a lot of incidents and mark them as related to a parent incident and then close the parent incident and the related ones will get closed automatically. This is a smart feature!

The problem with this is that there is a poorly written filter that executes on every save/modification of an incidents and runs a windows server process of cscript to build an XML-file. Microsoft is not well known for its efficient small footprint applications. So when the SM-guy modifies 50 incidents at the same time there will be 50 cscript processes running at the server and consuming all the available memory including some virtual memory on disk. And I can only guess what will happen when 50 identical cscript processes tries to update the same XML-file at the same time. You will probably get a lot of waiting... and helpdesk personnel that can not do it's work.