Performance testing with 10,000 concurrent users

Lately, I have been concentrating on performance testing within an application I am working on. This is using the Team test portion of Visual Studio. It certainly is a powerful tool and is exponentially  better than anything offered to date by Microsoft.

We have some requirements to ensure we hit around 5,000 concurrent users within this app, so pumping to 10,000 is a good way of ensuring that we can meet the goals and find any potential bottlenecks or issues.

Currently, I have all the test runs automated so that every night, a 10,000 concurrent user test is initiated that runs for approximately 12 hours. On a weekend, the automated test run, also tests using 10,000 concurrent users, but runs for approximaely 60 hours. The weekend test is identical to the overnight run except that it lasts longer, and has a slower ramp up to the 10,000 users.

Additionally, we have a web farm of 7 web servers, all connecting to a database server, and all these servers have a set of performance counters that are also scheduled to start and stop at the times of each test. These results get logged to a database, so that I can look through them the next day.

We are using a client machine farm of approximately 20 PC's all administered remotely through the Team test console, to generate the load of 10,000 concurrent users. Sometimes we use more if the machines are available. This is one of the cool aspects of team test, I can install the load agent on a large amount of machines and enlist only some of those machines by manually, or by specifying properties of each machine and only enlist the machines that contain a particular property or attribute.

I will digress into the details of the test rig setup and how I go about setting up the automation and collection of performance data on later posts. This is simply an initial post to paint the picture on what I am doing at the moment.

I will say the the ability to examine the test result data using the interactive graph capability within team test is awesome. The user interface is a little quirky and not entirely intuitive at times, but you get used to it.

5 Comments

  • Sounds like fun Glav! You've now become my resident perf testing guru.



  • Hi,

    did you use any particular document/manual/walk through in order to setup the whole environment?

  • Paul:

    We have been setting up a Perfoamance Test lab using the the Microsoft Team tools.
    The tools are pretty good, but more importantly, do not cost a fortune when scaling up to 1,000s of users.

    We have a dedicated T1 in house just for Performance Testing and are working with a local Data Center to see what we can do about getting some serious bandwidth when needed.

    My intertest in your work stems from the information you might have gathered on what facilities (bandwidth and computer power) are needed to simulate X virtual users.

    Our company is MobiusLabs (MobiusLabs.com). My e-mail address is awlasuk@mobiuslabs.com.

    If you, or any of your assicates would like to chat abut PT please contact me.

  • What is the "think" time for each user?
    Do you measure bottlenecks on your loading PCs?
    What counters are you recording?
    What is your calculations to convert those 10K to # of requests per second?
    Thanks.

  • That is awsome. Could you please share your script.

Comments have been disabled for this content.