Re: Capacity Planning / Hardware limits
- From: Kokkers <kokkers@xxxxxxxxx>
- Date: 2 May 2007 01:08:11 -0700
On Apr 30, 7:04 pm, Steve Dz <Steve D...@xxxxxxxxxxxxxxxxxxxxxxxxx>
wrote:
Hi everyone,
I'm trying to determine the hardware requirements for a large scale
streaming buildout. The goal is "up to" 30,000 concurrent viewers, with
about 200 live events per month. Up to 5 events could be 'live'
simultaneously. Thanks greatly for any info provided!
1) Is it typical to build out a 'cache farm' that users connect to (i.e.
having the cache servers broadcasting from a single remote publishing point)?
2) Using current server technology - dual/quad core CPUs and gigE NICs, lots
of RAM & disk (such as Dell 2950), what is the realistic concurrent capacity
of a server pushing out 300K streams? The Windows Media Services FAQ (http://www.microsoft.com/windows/windowsmedia/forpros/server/faq.aspx...)
describes 28k streams on a single 233 megahertz processor and 256 megabytes
(MB) of RAM. Not very useful information. Are there more recent benchmarks?
3) What difference would going 64 bit O/S provide?
4) Are there any 3rd party hardware solutions that increase capacities that
anyone has experience with? Do traditional hardware load balancers work for
streaming media servers as well (F5)
Thanks again.
Interesting topic, allow me to add some experience and questions of my
own.
First of all, you need to tell us if we are discussing live playout
webcasting or streaming static video's on demand.
Getting 30,000 concurrent viewers on a livestream will be alot easier
compared to serving 30,000 concurrent ondemand video views.
Getting 30,000 concurrent livestream viewers will be a matter of
determining maximum serverload and then add servers till you get where
you want to be.
Determining the maximum load of a server will be the hard part, I was
not able to find much accurate benchmarks on recent hardware.
Modern hardware has made a big leaps since the P2 CPU's used in the
old benchmarks though.
1) Caching 'on demand' video files will be an interesting topic of
discussion. Basically disk I/O performance will be your first
bottleneck when serving on demand video files.
As far as I know Windows Media Services has no out-of-the-box caching
or distribution mechanism built-in. You should be aware of the fact
that WMS will utilize 512MB of RAM at most for caching. Adding alot of
RAM to your servers is a waste of resources from what I have seen. I
would really like to know why Microsoft chose to only use 512MB of
system memory. WMS does provide support for custom 'caching' plugins,
I would love to hear what people have produced in this area. The SDK
supposedly has source examples availble.
I would really like to know if 64bit WMS can cache more than 512MB in
memory!
For performance and redundancy sake you will need to run multiple
servers. Are you going to use central storage (SAN) or direct attached
storage (DAS) to store your video files? You will either need a really
fast, large and stable (single point of failure) SAN or face the
problem of file replication to multiple Windows Media servers.
Depending on the pool of video's and how it is used, at any time 99%
of the requests could be for 1% of your data. Succesfully implementing
somesort of caching mechanism will greatly offload your spindles and
vastly increasy your capacity. You will want to run a minimum of 2
networks on your servers, one for server to server communication and
one to talk to your end-user.
I am really curious as how other people are handling caching.
2) Modern clocked multicore Xeon CPU's are fast and likely not a
bottleneck in your system. A modern server should be able to fully
saturate a Gigabit ethernet channel easily. We use dynamic link
aggregation on Intel multiport Gigabit Nics to create multi-gigabit
trunks to our core router (802.3AD).
We are still not sure where the limit is here but we have been able to
succesfully saturate nearly 2 Gbit per server during live events.
Microsoft has an application called 'Windows Media Load simulator'
available but stress testing multi gigabit links isnt that easy.
3) One of the downfalls of 32bit numbers is the limit to 4,294,967,296
in size (2^32). Windows stores the amount of memory in bytes, and
since a 32bit number can only hold 4,294,967,296, there is a limit of
roughly 4 billion bytes (or 4 gigabytes). Basically 32bit Windows can
not address more than 4Gigabyte of memory. Windows will segment the
available 4GB into 2GB of userspace memory and 2GB kernelspace memory.
I have seen WMS will not use over 512MB for file caching so this is no
problem. However, for each client connection WMS will need to reserver
some memory. If you want to run alot of concurrent connections it will
be possible for WMS to run out of available 2GB user space memory.
Some tweaks are possible like using Physical Address Extension and
dividing the kernelspace memory differently but my advice would be to
always use 64bit Windows. Driver or 3rd party application
incompatibility may be a problem though but this is easily checked.
4) In a 30,000 concurrent viewer setup you will want at least one load
balancer. Traditional loadbalancers are bad for streaming media since
they will want to route return traffic through the loadbalancer. On
high volume traffic this is either impossible or will require very
expensive network equipment. Foundry networks is known for its DSR or
Direct Server Return / Response but Cisco or other vendors will have a
similar solution.
Incoming requests are handled by the loadbalancer to a real server but
return traffic will flow directly back to the client requesting the
data.
It is hard to get some absolute numbers and limits in high volume
setups because alot of variables are involved.
You will want your platform scalable because tomorrow the 300Kbit
stream will be a 800Kbit stream served to 60,000 people.
Some of my first thoughts, would like to hear some others views.
.
- Follow-Ups:
- Re: Capacity Planning / Hardware limits
- From: Steve Dz
- Re: Capacity Planning / Hardware limits
- Prev by Date: Re: Media Server on 2003 SBS Premium (possible ISA issue)
- Next by Date: WMS9 server hang when using Unicast and Multicast is not work
- Previous by thread: Re: Media Server on 2003 SBS Premium (possible ISA issue)
- Next by thread: Re: Capacity Planning / Hardware limits
- Index(es):
Relevant Pages
|
