Peer To Peer Processing?
Anyone know of a tool / product / project that creates a Peer To Peer processing environment?
I’m not talking about batch processing like SETI@Home. I’m talking dynamic workload balancing across a self-discovering peer network of workers, with multiple interfaces for submitting working and monitoring progress.
I’m thinking something like Limewire for processing, rather than stealing IP from content creators. :-p
Anything out there? I thought this was the promise of peer-to-peer? That huge clusters of self-maintaining servers would be able to balance processing load…
Just asking, because I have a need for a such a beast.
February 25th, 2009 at 8:56 pm
um . . . is something like xgrid what you are looking for?
February 26th, 2009 at 1:13 pm
Very Similar, yes.
But here’s the problem with XGrid: Show me a corporate enterprise that has as many OS X servers in their racks as they do x86 boxes.
I want something that can run on multiple platforms, utilizing -all- the spare resources a company may have, not just their OS X boxes (if they have any), and I want it decentralized. I don’t want a single command node in the cluster. I want the cluster to be intelligent enough to not need a single point of failure.
Current client (as well as all past clients), don’t have a single OS X box on the premises. XGrid is for Academia (read Xserve clusters) and OS X software houses, where the vast majority of OS X servers are located.
February 27th, 2009 at 3:10 am
Bryan,
I know it’s not exactly what you’re asking for, but if you can find a way to run your processing in a compatible manner (LAMP, but I suspect you’re not doing that) in my search for my current business venture (not the one you’re aware of, as I need to get funding for that: this one is relatively light on funding requirements for startup) for web hosting, I found a web host provider that has something that does very much along the lines of what you’re looking for.
Look at http://mediatemple.net/webhosting/
March 2nd, 2009 at 10:37 am
It’s similar, although instead of on-demand hosting growth, I’m looking for on-demand processing growth. The idea is for maximum throughput of records processing, not request processing.
Similar concept, but the devil is in the details.
XGrid is clearly the closest thing to what I’m looking for, but it still has a single point of failure, dedicated master node. What a shame. They’re so close.
March 6th, 2009 at 2:14 am
Hey Bryan,
As an unexpected side-effect of looking for other things, I ran into this, which I think you should look at:
http://www.tsunamictechnologies.com/services.htm
This is definitely closer to what you’re looking for: you can run pretty much anything, it sounds like (as long as it can somehow run under Linux) and you can at least choose how many machines it runs on. I’ll leave the devil of the details for you to explore with their no obligations account, sales critters, etc.