Return-Path: Received: from atl4mhob15.myregisteredsite.com ([209.17.115.53] verified) by media-motion.tv (CommuniGate Pro SMTP 4.2.10) with ESMTP id 4897216 for AE-List@media-motion.tv; Mon, 12 Nov 2012 21:54:15 +0100 Received: from mailpod.hostingplatform.com (mail.networksolutionsemail.com [205.178.146.50]) by atl4mhob15.myregisteredsite.com (8.14.4/8.14.4) with ESMTP id qACKvRJ3002252 for ; Mon, 12 Nov 2012 15:57:27 -0500 Received: (qmail 19866 invoked by uid 0); 12 Nov 2012 20:57:27 -0000 Received: from unknown (HELO ?192.168.0.3?) (dsd@darkskydigital.com@24.73.238.28) by 0 with ESMTPA; 12 Nov 2012 20:57:27 -0000 From: Dann Stubbs Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: multipart/alternative; boundary=Apple-Mail-38-784492571 Subject: Re: [AE] Semi OT: Yet another mac mini render farm thread Date: Mon, 12 Nov 2012 15:57:27 -0500 In-Reply-To: To: "After Effects Mail List" References: Message-Id: X-Mailer: Apple Mail (2.1084) --Apple-Mail-38-784492571 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 hey tony, (and sent to the AE list for benefit of others wondering) if you go with mac mini's try to get a few spare external power = supplies from apple parts to have on hand as if they burn out / go dead = - that would be something to replace right away with not waiting on = replacement swap times. (also check out the prices and how much they may = cost before you decide. one reason i never looked at Dell was they used = proprietary power supply connectors on their motherboards - that means = only their PS can be used and at their whim for price and supply of = them) as for heat - i don't know how much cooler mac mini's could be then a = normal PC heat of i7 - heat is heat - it has to go somewhere...once i = had the main AC go out a couple years ago and my studio with the farm it = it went over 105 degrees in less then 5 minutes (from high 60-low 70's) = - i've actually talked to my wife about moving to alaska or somewhere = cold where all the heat could be of use- like i could exchange heating = the office building for my rent or something.. would never have to pay = to heat the studio in a cold climate - where in tropical florida i'm = paying year round to cool instead. wife says NO to snow, so stuck for = now it seems. : ( network gear - yeah they get hot too with heavy use and use quality ones = as the power supplies and network routers themselves will melt/fry. been = there done that when starting up 10 years ago and now only metal pro = grade network stuff...=20 i've thought about using the mini's too time to time - but just afraid = of how 24/7 would handle it... i have friends with web hosting that use = them but they do not sustain 100% processor usage for days on end - web = hosting is not anywhere as demanding as rendering - so they get hot - = not sure if they can handle it (or if the power supplies can)- i once = tried some "green" PC power supplies - more efficient types and they all = were dead in 60 days - most died in the first 30 days - as the power = saving components were not meant to be 24/7 it seems - i won't make that = mistake ever again. commercial grade power lines - yeah - many years ago when i was first = starting out and tripping circuits under heavy draw (long past those = days) i had to have heavy commercial grade lines put in - expensive and = you will need more then you think. check the AMP load of one render node = at 100% (they sell meters and stuff for that) now look at what the line = can pull - multiply and get out the checkbook. years ago the electrician = showed me what burned power lines in the wall look like - it's scary or = was to me... the insulation was dried and flakey, brown/black like the = outside of a burned marshmallow. watch those power limits per line. i = routinely test my power pull on each commercial grade line now and have = a patch cut out of the wall where i can touch the wires inside the wall = run to see how hot they feel. don't underestimate the power bills... summer has my electric bills for = the studio now at $1600 per month - yeah... ouch. my electric bill = comes in a special gold envelope. (kidding, but it should) the coming = EPA regulations are going to shut down a lot of coal electrical plants - = so with no new nuclear or other plants taking up demand - i don't know = what is going to happen in the next couple years for electric rates... = solar and wind aren't going to add anything to 24/7 demand needs. this = year along i have seen a 25% increase and it gets worse as you are a = high "user" of electric you keep going over the limits they have in = tiers and into higher and higher KW rates. i.e. first 1000KW is one = price, then next 1000KW is priced higher etc, then the next etc... i've = had heavy use months go 50% above what the bill was last year. and the = real big increases in power costs aren't even here yet. (never did i = think politics and EPA regulations would affect my business but it is = now) UPS's are a must - and i just spent $6,000 on upgrades to mine (again)- = need another $4,000K to finish the upgrade still... trust me the cheap = ones won't last, the prosumer ones last a bit longer, but replacing = batteries is expensive and the consumer types don't seem to handle it - = i spent $4k last year on some "prosumer" ones to try them out and 1 year = later 80% of them were dead... another waste of money... the APC 1500 = ones (tall consumer/professional workstation types around $200) seem to = last one to two years, but that's it and replacing batteries are just = about as much as buying new ones.=20 just have to bite the bullet and go with the commercial grade and that = is the expensive ones if you want them to last. usually i have to = replace them every two years - so that money adds up too. the ones i = get now are $600 each... hope to get two or maybe fingers crossed three = solid years from them... but we'll see. the only other reason to be careful with mac's is the software - lots of = the c4d plugins (and other plugins are PC only) for your own studio this = should be ok i suppose as if you are using macs you should have matching = plugs for a mac farm ok. good luck, let me know if you have any questions, dann ___________________________________________________________________ Dann Stubbs - dann@darkskydigital.com Dark Sky Digital - http://www.darkskydigital.com http://www.RenderKing.com Value Priced C4D, VRAY Render Farm ___________________________________________________________________ On Nov 12, 2012, at 3:19 PM, Tony Romain wrote: > I think where I'm at is trying to figure out what the main limiting = factor is here versus building something from the ground up, piece by = piece (probably PC based for cost). >=20 > Do you have a sense of specifically what the networking issues were? = For cinema rendering, my understanding is that once the net render files = have been distributed amongst the nodes, their really isn't a ton of = network traffic, except for the rendered frames being copied back over = to the server. However, I may be wrong on this=85 I don't know a ton of = what goes on under the hood of these farms=85 >=20 > Have been exchanging emails with the owner of this company: >=20 > http://h-sq.com/products/minirack/index.html >=20 > to see if he's heard any anecdotal info from any of his clients. He = hand't heard anything specific other than describing one server set-up = he'd heard of that utilized 160 mac minis for some type of test = automation (not sure what that is), but they were all being taxed 100% = of the time and seemed to run fine=85 >=20 >=20 > -- > tony romain | principal/creative director >=20 > trance > motion graphic animation and design > 323 651 1114 > www.trancedesigns.com >=20 > From: Teddy Gage > Reply-To: AE list > Date: Monday, November 12, 2012 12:05 PM > To: AE list > Subject: Re: [AE] Semi OT: Yet another mac mini render farm thread >=20 > My main concern would be the networking back end. Unless you have an = applescript wizard on hand you may run into some problems, even with a = networked rendering package like deadline. We tried to do something like = this at a place I was working a few years ago and it was rage-inducing. = The frustrating thing was "it should have worked" but it just didn't. = There were always problems. maybe if you installed win 7 pro in bootcamp = on all of them. but I guess that would defeat the purpose. Things have = probably come along since then however --Apple-Mail-38-784492571 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=windows-1252 hey = tony, (and sent to the AE list for benefit of others = wondering)

if you go with mac mini's  try to get = a few spare external power supplies from apple parts to have on hand as = if they burn out / go dead - that would be something to replace right = away with not waiting on replacement swap times. (also check out the = prices and how much they may cost before you decide. one reason i never = looked at Dell was they used proprietary power supply connectors on = their motherboards - that means only their PS can be used and at their = whim for price and supply of them)

as for heat = - i don't know how much cooler mac mini's could be then a normal PC heat = of i7 - heat is heat - it has to go somewhere...once i had the main AC = go out a couple years ago and my studio with the farm it it went over = 105 degrees in less then 5 minutes (from high 60-low 70's)  - i've = actually talked to my wife about moving to alaska or somewhere cold = where all the heat could be of use- like i could exchange heating the = office building for my rent or something.. would never have to pay to = heat the studio in a cold climate - where in tropical florida i'm paying = year round to cool instead. wife says NO to snow, so stuck for now it = seems. : (

network gear - yeah they get hot too = with heavy use and use quality ones as the power supplies and network = routers themselves will melt/fry. been there done that when starting up = 10 years ago and now only metal pro grade network = stuff... 

i've thought about using the = mini's too time to time - but just afraid of how 24/7 would handle it... = i have friends with web hosting that use them but they do not sustain = 100% processor usage for days on end - web hosting is not anywhere as = demanding as rendering - so they get hot - not sure if they can handle = it (or if the power supplies can)- i once tried some "green" PC power = supplies - more efficient types and they all were dead in 60 days - most = died in the first 30 days - as the power saving components were not = meant to be 24/7 it seems - i won't make that mistake ever = again.

commercial grade power lines - yeah - = many years ago when i was first starting out and tripping circuits under = heavy draw (long past those days) i had to have heavy commercial grade = lines put in - expensive and you will need more then you think. check = the AMP load of one render node at 100% (they sell meters and stuff for = that) now look at what the line can pull - multiply and get out the = checkbook. years ago the electrician showed me what burned power lines = in the wall look like - it's scary or was to me... the insulation was = dried and flakey, brown/black like the outside of a burned marshmallow. = watch those power limits per line. i routinely test my power pull on = each commercial grade line now and have a patch cut out of the wall = where i can touch the wires inside the wall run to see how hot they = feel.

don't underestimate the power bills... = summer has my electric bills for the studio now at $1600 per month - = yeah... ouch.  my electric bill comes in a special gold = envelope. (kidding, but it should) the coming EPA regulations are = going to shut down a lot of coal electrical plants - so with no new = nuclear or other plants taking up demand - i don't know what is going to = happen in the next couple years for electric rates... solar and wind = aren't going to add anything to 24/7 demand needs. this year along i = have seen a 25% increase and it gets worse as you are a high "user" of = electric you keep going over the limits they have in tiers and into = higher and higher KW rates.  i.e. first 1000KW is one price, then = next 1000KW is priced higher etc, then the next etc... i've had heavy = use months go 50% above what the bill was last year. and the real big = increases in power costs aren't even here yet. (never did i think = politics and EPA regulations would affect my business but it is = now)

UPS's are a must - and i just spent $6,000 = on upgrades to mine (again)- need another $4,000K to finish the upgrade = still... trust me the cheap ones won't last, the prosumer ones last a = bit longer, but replacing batteries is expensive and the consumer types = don't seem to handle it - i spent $4k last year on some "prosumer" ones = to try them out and 1 year later 80% of them were dead... another waste = of money... the APC 1500 ones (tall consumer/professional = workstation types around $200) seem to last one to two years, but that's = it and replacing batteries are just about as much as buying new = ones. 

just have to bite the bullet and go = with the commercial grade and that is the expensive ones if you want = them to last. usually i have to replace them every two years - so that = money adds up too.  the ones i get now are $600 each... hope to get = two or maybe fingers crossed three solid years from them... but we'll = see.

the only other reason to be careful with = mac's is the software - lots of the c4d plugins (and other plugins are = PC only) for your own studio this should be ok i suppose as if you are = using macs you should have matching plugs for a mac farm = ok.

good luck, let me know if you have any = questions,

dann


Dann = Stubbs  -  dann@darkskydigital.com
<= div style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; = margin-left: 0px; ">Dark Sky Digital -  http://www.darkskydigital.com

http://www.RenderKing.com &nbs= p;Value Priced C4D, VRAY Render Farm



On Nov 12, 2012, at 3:19 PM, Tony Romain wrote:

I think where = I'm at is trying to figure out what the main limiting factor is here = versus building something from the ground up, piece by piece (probably = PC based for cost).

Do you have a sense of = specifically what the networking issues were?  For cinema = rendering, my understanding is that once the net render files have been = distributed amongst the nodes, their really isn't a ton of network = traffic, except for the rendered frames being copied back over to the = server.  However, I may be wrong on this=85 I don't know a ton of = what goes on under the hood of these = farms=85

Have been exchanging emails with the = owner of this company:


to see if he's = heard any anecdotal info from any of his clients.  He hand't heard = anything specific other than describing one server set-up he'd heard of = that utilized 160 mac minis for some type of test automation (not sure = what that is), but they were all being taxed 100% of the time and seemed = to run fine=85


--
tony romain
trance
motion graphic animation = and design
323 651 = 1114

From: Teddy Gage <teddygage@gmail.com>
Reply-To: AE list <AE-List@media-motion.tv>Date: Monday, November 12, = 2012 12:05 PM
To: AE list = <AE-List@media-motion.tv>Subject: Re: [AE] Semi OT: Yet = another mac mini render farm thread

My main concern would be the networking back end. Unless you = have an applescript wizard on hand you may run into some problems, even = with a networked rendering package like deadline. We tried to do = something like this at a place I was working a few years ago and it was = rage-inducing. The frustrating thing was "it should have worked" but it = just didn't. There were always problems. maybe if you installed win 7 = pro in bootcamp on all of them. but I guess that would defeat the = purpose. Things have probably come along since then however

= --Apple-Mail-38-784492571--