Handling 1.5 Billion Page Views Per Day Using ASP.NET 2.0

One of the highlights for me at the MIX conference earlier this week was being able to chat with customers about the success they’ve had with sites they’ve built on top of ASP.NET 2.0 and IIS 6.   

 

MySpace.com was definitely the biggest highlight.  For those that aren’t familiar with MySpace.com, it is the fastest growing site on the Internet right now.  They have 65 million registered subscribers, and are registering 260,000 new users each day.  According to the Media Metrix report (an independent analyst firm) MySpace.com had more page views in February than all of the MSN and Google sites combined.  That is some serious load.

 

They re-built and re-deployed their site on ASP.NET 2.0 shortly after we shipped last year.  Some of the pretty amazing statistics Aber and Allen (the MySpace CTP and VP of Engineering who were both in BillG’s keynote session) shared at MIX about the MySpace.com site:

  • MySpace.com is now processing 1.5 Billion page views per day
  • MySpace.com handles 2.3 million concurrent users during the day
  • MySpace.com’s average server CPU utilization went from 85% to 27% after moving (from another technology) to ASP.NET 2.0

Those lucky enough to attend MIX (or watch the web-casts online) were also able to see Allen demo the new Ajax Profile Personalization engine that MySpace is building using Atlas (which you can learn more about here – including watching a video by me building a to-do task list app from scratch with it), as well as attend the awesome “Building Mega-sites” talk that MySpace gave together with folks from Microsoft.com about real-world, super large-scale, development and deployment experiences.

 

Other great real-world Atlas deployments by customers that were highlighted at the MIX event included Squeet.com, TitleZ.com, and PageFlakes.com (you can click on the “showcase” tab on http://atlas.asp.net to read more about them and their experiences). 

 

It is hearing stories like these that keeps me motivated and having fun. :-)

 

- Scott

 

P.S. In case you were wondering, the top-6 domains in terms of page-views in February according to Media Metrix were: 1) Yahoo, 2) MySpace, 3) MSN, 4) Ebay, 5) Google, and 6) Hotmail.

 

P.P.S. 4 of the top 6 sites (MySpace, MSN, Ebay and Hotmail) run on IIS and Windows. J

 

43 Comments

  • I was wondering who the #1 site was...figured it was Y! (I'm guessing it's all the Y! groups that generate so much traffic). I bet MySpace will take them over at some point though.



    MySpace really needs to get of those nasty CFM querystrings though. I was amazed at how young those guys were.

  • Um...



    There is still an awful lot of .cfm (Coldfusion) on Myspace!









  • Most of the CFML has been moved to BlueDragon.NET for now, which is a product which allows you to run the ColdFusion language using a HTTPRequest handler from within the .NET framework. This was done to leverage the existing CFML codebase and take advantage of the performance increases Scott mentioned, while they migrate the application completely to ASP.NET

  • I just checked the myspace site and it appears to be using ColdFusion... Where does it use ASP.Net 2.0?

  • Hi Onovotny,



    Some parts of the site like Browse are all .aspx extensions. As Jeff mentioned above, other parts of the site still have a .cfm extension, some of which is mapped to BlueDragon.Net -- which is an ASP.NET IHttpHandler that parses and can run ColdFusion syntax (but which is still running on ASP.NET). The backend cache servers are also all ASP.NET based.



    Hope this helps,



    Scott

  • I realize that they are mapping the CFM requests to asp.net, but this could all be handled on the server and MySpace could still dish up .aspx pages (well, pages w/ that extension at least) and the user wouldn't even know the difference. Write an httpHandler that intercepts all requests for CFM pages, intercept the querystring vars, map those requests to the appropriate .aspx page (along with a 302 response for a perm redirect), and be done with it.



    I do realize the mappings are complex, but all asp.net credibility is lost when folks still see a .cfm file extension. They can't keep doing this forever.

  • Hi everyone,



    I work on the MySpace C# codebase...



    To clarify, we wrote a custom configuration section that maps "fuseaction" URL parameters to ASPX extensions so that we'd maintain link integrity. The only place we aren't doing this is 'Browse' and certain other new features. Meanwhile, as Scott said the parts of the site that are running in ColdFusion are essentially doing so in ASP.NET 2.0 (via BlueDragon).



    Thanks for the mention, Scott. It's been an exciting time putting this together and I can't imagine pulling this off on another platform.



    Chris



  • Totaljobs.com also uses asp.net, it's the biggest recruitment site in the uk: well impressed by it.

  • MySpace.com’s average server CPU utilization went from 85% to 27% after moving (from another technology) to ASP.NET 2.0



    What was the "other" technology?

  • Yeah thanks for the info. We are still having some issues at our company because the java people still think that .net is not for enterprise development. Besides our teams showing how our banks are switching over to .NET, this might be the silver bullet we can use to convince the sceptics.

  • How many servers does it take ti run myspace?

  • Care to indicate something about the hardware the system is deployed on?

  • Hi Bob/Thozie,



    I'll need to let the MySpace folks comment on that since I'm not sure how much they can share (Chris from MySpace already posted above).



    Given the load (and the fact that there are a lot of non-page-view downloads like images + videos + music), I was actually surprised that they didn't need more servers.



    Hope this helps,



    Scott

  • now Scott that you've mentioned MySpace.com in your blog there is a some possibility that they got even more traffic and eventually go to 'slashdotted' state =)



    /me crossing fingers ASP.NET2.0+IIS6 will handle it <g>

  • Hi Scott,



    Parts of Yahoo.com are running on IIS too. So 5/6 :)

  • Hi Max,



    I didn't realize that -- very cool! There are actually a few google sites that are running using ASP.NET as well, but I wasn't counting them (or the Yahoo ones) yet since their main sites are still running something else.



    We are still trying to convince them to convert though. :-)



    Thanks,



    Scott

  • Is it possible to get some more details on how myspace.com is designed? More details on architecture. How are you doing load balancing and how do you optimize database access, etc..



    Or do any of you have links to articles with info about other high traffic real world web apps?

  • I thought eBay had moved to Java - they have a Powered by Java logo on their homepage.



    quote:

    "eBay is Java™ Powered running on powerful Sun Fire UltraSPARC® and AMD Opteron™ servers supported by Sun services and solutions."

  • Hi Rich,



    I believe EBay is using SUN servers for their database and parts of their middle-tier. But their web-front end is running on IIS and Windows (if you go to www.netcraft.com you can verify this for any domain).



    Hope this helps,



    Scott

  • Which is the biggest sql server and the rest in .net(com+, ws, asp.net ...) site that you know? a pure ms site.



    Do you have statistics and the architecture?

  • Scott, you mentioned the web-casts online. Is that still available?

  • Hi Scott,

    Does Myspace.com use whole Microsoft's solutions for there whole system?

    ASP.NET 2.0 as the front-end,

    .NET (may be COM+ or other) as Middle-tier,

    Microsoft Sql server 200x as Database-tier.



    Am I right?

  • Hi Pablo/Robinz,



    MySpace uses SQL on the backend for storage. There are a lot of other large sites I can think of -- Dell.com being one of the heaviest used ones (all their orders go through an ASP.NET system).



    Hope this helps,



    Scott

  • Hi Ryan,



    I believe the Mix06.com site is planning on posting webcasts of the talks in a few weeks. Keep an eye on the site and hopefully we'll see some soon.



    Thanks,



    Scott

  • Where can we download the presentation for Building Mega-sites?

  • scott, I have the same questions as Pablo Castilla about the number of servers.



    To handle so many concurrent users, aren't they using the asynchronous request and is it the reason that reduces the cpu usage?



    Wondering how many concurrent maximum users can be in .net 2.0, both synchronous request and asynchronous, and what are the parameters to optimize.



  • There are interesting points but I have 3 comments for anyone reading this and thinking ColdFusion was a limiting factor for the site.

    1. The cfm code is running on BlueDragon right now but was the old version of MySpace on BlueDragon or Allaire/Macromedia/Adobe ColdFusion? Was it running on top of a J2EE Server on Enterprise ColdFusion or was it standalone? There are 4 versions of ColdFusion right there. I'm not sure how each version of Adobe ColdFusion stacks up against BlueDragon but there are white papers outlining the key differences in performances.

    2. Poor code is poor code in any language. You don't port your coding mistakes typically and hence make improvements in a relaunch. These improvements would probably also include new hardware and some analysis of the bottlenecks that weren't ironed out of the old version.

    3. If you hire M$ programmers to write CF code they will typically write bad CF code (through a lack of experience) and insist that a M$ port will fix it. I have experienced this myself. Similarly, if I wrote M$ code Im sure it would suck and then I would insist on a port to CF :-)



    To draw a parallel, we developed and manage the online registration application for the Chicago park district (the world's largest park district). We recently rewrote the registration portion as an AJAX application and reconfigured the hardware. During registration we experienced 500 completed orders in the first minute operating on four load balanced ColdFusion 7 Servers running IIS and connecting to a SQL Server database.



    In short it is impossible to directly draw the line and say ColdFusion was why it ran slowly. Hands up those of you who ever rewrote an application and made it slower and kept your job?

  • Hi Rich,



    Their dynamic UI is still a custom IIS ISAPI extension (look at the URLs and the http headers that come back). ;-)



    Thanks,



    Scott

  • Hi Travis,



    I believe they will be posting the webcast on the mix06.com site soon.



    Thanks,



    Scott

  • I could have swore I read ebay was running sql server as their backend, did they switch some time back?

  • I'd like to know what type of disk array storage an configuration they are using for their database. Like an EMC CX400, how fast the storage processors are, how much RAM the storage array has, how fast (RPM) are the hard drives, what RAID configuration is being used, and how many physical drives per LUN.

  • To say that it's 1.5 billion "page" views is extremely misleading. Most of those server requests are to x.myspace.com for static accessory content such as CSS, JS and images. I like ASP.NET too but touting 1.5 billion "page" views is just marketing drivel.

  • Hi P,



    Actually -- apprently the 1.5 Billion number is page views, and doesn't count images or CSS or other static files.



    Hope this helps,



    Scott

  • I stand corrected. Those numbers are just so eye-popping that the skeptic in me took over. Sorry to sound accusational. Great job by the way!



    P

  • people2pray is a hot idea -- i wish you the best with that



    I have a domain i want to make into a social networking and more site, anyone know of myspace style scripts for sale that can run on windows2k3 server?



    the domain is GlorySpace.com and will be for christian youth.

  • Im glad to see BlueDragon getting the attention they deserve. Those guys have been busting their ass for quite a while to build a great CFML processor, and best of all you can get it for free.

    Good job Ole Blue.

  • Blue Dragon .NET is $3000 for a single processor server and $6000 for a two processor server. I'm not sure what your definition of free is but it seems to be different from mine.

  • I'd sure love to see a well done tutorial with the above title.

    Can anyone point to any reference material that discusses the programming and database approach to building a social networking site? I'm building one now and I could really use the help, instead of working in the dark as I am now.

  • Why is it so hard to believe the results of real-world, HEAVILY trafficed sites saying they are using MS technology? Seems like a fair number of people can't get past their hate of company A's technologies to realize they are just spouting for company B's. Productivity benefits of Cold Fusion? Have you got your head around asp.net and coding in C# or VB.net? SO clean, and very fast.

    My experiences in efficiency with .net are similar. End-to-end, the technology makes a difference and proves itself. I don't care who is selling it. It works (fast) and that's job one. The languages supported are easy to learn and there's no piecing together of 4-10 different products to get sites together. The tools given to you out of the box get you going fast. PLUS, the development tools themselves are tight and the code all ends up clean...

    The simple fact that myspace has the massive traffic it does and moved to .net really means enough in itself. Maybe a few years from now they'll be crying about how they wish they never made the move... and Rupert Murdoch will be broke. ehhhh Not likely. Not good, but not likely.

  • This by far is the best implementation example I can say in the real world for massive traffic sites regarding ASP.net; I can hardly wait to see what else MySpace developers poor into the project. With the AJAX implementations, new .net technologies are gonna make Myspace the defacto... (I think it already is, but hey.. ya never know when yur little blog site might become the next one..) I guess it could be compared to some others out there that made huge improvements by moving to .net (ie. quixtar.com).

  • My site Boomtrek.com is running .NET 1.1 so far so great... getting 1000 hits/month, not that much but upgrading to C#.NET 2.0 to handle future growth after seeing the new great features introduced. I still think MapPoint.NET Web Services should be more reasonable to buy at $8500/year it's a little ridiculous since GoogleMaps are free.

  • How about PHP?

    Does anyone think that PHP can challenge ASP.NET in term of efficiency, scaling and performance?

    I know friendster is coded using php though. But it's not as big as myspace...

  • wow thats great.
    with .net the resources are managed efficiantly.

Comments have been disabled for this content.