Sunday, September 6, 2009

Slowing down your server process does have some benefits

Although lot people are finding various ways to speed up their application servers to improve performance, there are also benefits to slow down your server process especially when testing it to find problems that you won't see in normal scenarios. As you might have seen in your experience that 90% of the time things work well in a normal scenario when there the load on the server is normal, db response throughput is normal, network throughput is optimal , etc. But all of a sudden things start to break when something gets backed up like db fails to respond in an expected time leading to number of concurrent requests due to browser refresh by the user might break the applications due to deadlock or other concurrent related problems. Due to the availability of powerful multiprocessor severs nowadays , requests are executed in terms of milliseconds , hence it would be difficult to catch concurrency related problems unless you recreate the scenario where the requests are taking more time to process. Although one might say this can be simulated by doing a load test on the server, i agree but if there is a problem it would be a overwhelming task and extremely difficult for someone to debug given the amount of requests and objects created during the test, especially if there is a memory leak. Hence the same effect can be achieved by slowing down your server process with just using few concurrent requests which makes things easy to debug and trace.

For (e.g) check this below jsp lrumap.jsp, where it creates a simple LRUMap object, puts it into the session and then serializes it. You will not find any problems with the jsp as long as it runs quickly (usually takes about 10ms is a decent desktop server) with multiple concurrent requests, but if the code slows down due to I/O or some other reason , you will end up with the java.utl.ConcurrentModificationException if the code is executing the serialization part so.writeObject(lmap); and adding of the element lmap.put(lobj[i],new Integer(i)); at the same time even though the LRUMap is synchronizedm, the reason being the serialization code uses an iterator to iterate through the elements in which the iterator is not synchronized which caused the exception. Unless you make the server process to run slow you will never end up catching it until you see it in production during high load conditions where it will become extremely difficult to debug.

How to slow down a process ?

The trick is that you basically limit the CPU usage of the process which will slow down the process as it won't get enough CPU cycles to execute. There is a open source program in Linux CPU Limit can be used for this purpose or you can write a simple shell script where you can send STOP and CONT signal in a loop using the kill command. Although the later is not efficient but it works, but the CPU limit program is much precise even though it'd doing the same but through a C program.

1) Download and install the CPU Limit program.
2) If your System Administrator don't allow then use the shell script,

while [ 1 ]
kill -STOP $1
usleep 100000
kill -CONT $1

Note using usleep can make the sleep in microseconds where as sleep will sleep in seconds which is not suitable for process executing transactions in milliseconds.

3) find the process id of the process you wanted to slow down.

4) Execute cpulimit -p -l

(e.g) cpulimit -p 11502 -l 10 - this command will allocate max of 10% CPU to the process 11502.

No comments: