scratchme: Do you mind if I privmsg you some questions?
scratchme: (I know the asking to ask, but maybe you don't care to pm at the moment)
quux: sure, go for it
scratchme: I got this idea from the seminar I went to.
scratchme: I don't have anywhere near the bucks they want.
scratchme: And it seems to me that it should be possible to make a poor man's version of what they got using iSCSI targets on Solaris using ZFS + 2K worth of hardware.
quux: Sure. openfiler.org
scratchme: Then using VMWare server on multiple boxes you could have scripted, or manual fail-over since the entire VM setup including host OS would be on the iSCSI LUN.
quux: no ZFS at this time ... but 90% of the functionality anyway
scratchme: If your VM instances had their own LUN, you could resume a failed VM in another VMWare server.
scratchme: Or does this sound crazy?
quux: Well, hmm.
quux: First, the only failure this really recovers from is failure of iSCSI connectivity
scratchme: I'm talking trying to take say less than $10k and make a fairly resilient server setup which the other guys can beat, but they end up charging you 6 digits.
quux: or perhaps an overloaded vmware host
scratchme: Yes, or if you had iSCSI boot VMHosts, if the say fans die, the host might overheat and shutdown.
scratchme: Or faulty memory or whatever.
quux: Actually I'm doing something similar (but different) here at home. And aside from $4k in hardware, all no cost
scratchme: So you could have the other machine take over no problems.
quux: Lemme walk through it
quux: First, the hardware is 2x servers each with 2g RAM, 2TB disk, Athlon XP 64 3800 CPUs
quux: that was about $4k, 8 months ago
scratchme: Still not too bad these days.
quux: Then, software wise, each server runs FC4 and VMware Server
quux: Now, as vm guests, each one hosts an AD DC, and a windows fileserver
quux: the fileservers replicate their files back and forth via Windows DFS/FRS. So they are totally fault tolerant. If one goes down, users do not notice
quux: they also have 'shadow copies' taken every two hours (and staggered), so there is always a way to return to your file as it was 1 hour in the past
quux: the DC's are of course replicating via AD replication, so if one goes down, the other soldiers on
quux: Now I also have a couple of vm guests that are application specific. One windows, one linux
quux: There is only a single instance of each of these, but every hour I have a cron job that rsync's the underlying vmdk disk files to the other vmware host
scratchme: Now rsync does delta transfers?
quux: I can take one of them down, and bring it back up again on the other vmware server, within 3 minutes, losng no more than an hour's data
quux: right, but vmware disks break the data into 2gb chucnks, so the deltas are pretty large
scratchme: Doesn't all this have a lot of overhead?
quux: Still, this has been working out GREAT, and would be a very robust small office solution
quux: Or, with a third server as an iSCSI target that did snapshots, a more-robust 3-host solution
quux: Not really, once it's setup. Probably less overall network overhead than the iSCSI solution
quux: and, with a second NIC in each host, I could vlan that overhead
scratchme: Okay now what would you do if say you decided you needed bigger disks.
scratchme: How would you upgrade?
quux: I would simply add disks (I left room in the server cases for that) and expand the LVM array
scratchme: So you are running RAID 5?
quux: Nope. Just LVM in what would be the equivalent of RAID o
quux: no need for RAID, because the servers themselves are kind of like a RAID1. All data is always replicated to a second server
scratchme: See, this is partly why I came up with the idea I did.
scratchme: I have an old boss of mine that is interested in sharing a cabinet at a datacenter.
quux: Also, I defined really, really big disks in vmware itself. That much *phsyical* space is not there, but the windows and linux vms will always see this huge amount of space, and expand naturally into it
scratchme: If I could convince him when he does it to put in 1 server system between the two of us that's actually got some redundancy beyond just a RAID 1 mirrored disk, then I think it would be really good.
scratchme: So my tenative thought was 1-2 iSCSI rackmounts, and 3 VMWare hosts.
quux: right.
scratchme: So let's say 1 host would be my primary stuff, and the other 2 his.
quux: Actually, using ISCI target, I'd want two iSCSI hosts
scratchme: But if mine went down, it could come right back up on his, and vice versa.
quux: those would cluster in such a way that the ddata always sits on two servers
scratchme: Yeah, I was thinking of getting 2 iSCSI targets and clustering them.
scratchme: From what I'm reading ZFS can do this.
quux: Yah
scratchme: I'm sure other stuff can too.
quux: See, imho, we're to the point where a RAID set sitting locally in a server, is more reliable than the rest of the server
quux: which just moves the single point of failure to the server itself
quux: And I have seen FAR too many problems moving raidsets from one server to another in data-down situations
scratchme: data-down ?
quux: power/raid card/nic/mobo failure
quux: or whatever
scratchme: oh, okay.
scratchme: Well, honestly, I've never had a NIC die in use.
scratchme: Maybe I'm just lucky so far.
quux: So I say, screw RAID. Servers are cheap now. RAIS!
scratchme: Well, so let's say you decided you needed more capacity?
scratchme: Would you always add servers in pairs?
quux: Now you have continuous online, and you've moved the spof to the switch - and those are damn reliable
scratchme: A switch is also easy to replace.
scratchme: Any dummy paid $10/hr can fix that problem.
quux: Well, in this scenario, there are two serverbased bottlenecks: CPU, and RAM
quux: so yeah, if I needed more capacity, more servers. but not necessarily in pairs
quux: seems to me you could hopscotch one server at a time
scratchme: hopscotch?
scratchme: (I know the game)
quux: with two servers, each must have about 50% spare capacity to be able to take full load if/when the other goes down
scratchme: Yes.
quux: with 3 servers, each would need only about 33% spare capacity to redistribute load when one went down. And so on
scratchme: But what about replication?
scratchme: Wouldn't you need to replicate the data in every server to every other server?
quux: Nope.
quux: Each server only need replicate to one other
scratchme: Oh, I see what you mean.
scratchme: 1 -> 2 -> 3 -> 1
scratchme: Like that.
quux: right
scratchme: Okay.
quux: as your cloud gets larger, you need to plan for multiple servers going down at once, but I guess that would be manageable
scratchme: So if 2 dies, there's no replication of #1 anymore, unless you could setup alternate replication.
scratchme: Hrm.
scratchme: Interesting idea.
quux: beyond, say, 6 servers, it would probably make the best sense to have 2x monster fileservers
scratchme: Now, it doesn't bother you that the disk management is per server?
quux: or 3x - depends. because the new bottleneck there becomes, bandwidth
quux: At this small scale, no. In a larger scenario, I'd be using NAS
scratchme: Well, to a point bandwidth can be addressed by adding switches or buying better NICs etc.
quux: or iSCSI
quux: (I am not a fan of fiberchannel SAM)
quux: s/SAM/SAN
scratchme: When I looked at FC, I wasn't really interested.
quux: right. it never really made much sense, imho
quux: it's ok as a bus in the array
scratchme: Servers need less touchy feely management IMO.
quux: but building a separate, proprietary network seems silly to me and always has
scratchme: Seems it would make more sense to make multi-port NICs and team on gigabit switches or go to 10Gbit switches when they come out.
quux: there's never been a time when NAS couldn't do 80-90% of the performance SAN offered - at 50% the price
quux: yup
quux: although you have to be careful with muliport nics
quux: pick your switches *very* carefully
quux: the question is how loadbalancing is acheived
scratchme: Well, your setup sounds quite good, I'm just trying to figure out if I have the patience for it. My solution was rather simple, but didn't have as much automated fail over.
quux: really, the hardest part was getting SATA to work with linux and LVM
scratchme: Infact other than the RAID, and ZFS clustering (assuming I could figure it out), there wouldn't be any at the application level.
quux: after that it was just a few batchfiles
quux: and/or using standard WIndows file replication tech
scratchme: Well, maybe I'm lucky, but my server rarely goes down.
quux: and it worked flawlessly, btw. I had a disk faile in one server - and zero dataloss. My wife never noticed
scratchme: But, I'm sure that won't last forever.
scratchme: ATM, the server is in LA, and I'm 2600 miles away.
scratchme: heh
quux: re: multiport nics
quux: this is something most people don't realize until they attempt it
quux: adding another NIC port and teaming, often does NOT double your bandwidth
scratchme: Well, what it does do is allow you to connect using another IP though, and get extra bandwidth.
quux: say you have two servers on one switch. And repl files between the two ...
scratchme: So say you had 2 LUNs you could have 1 LUN each port.
quux: adding another NIC and teaming them, doesn't speed it up at all
quux: because of the way most switches handle teaming, all traffic ends up on one nic!
scratchme: That's unmanaged stuff right?
quux: well I should be more accurate there. I've experienced this on Cisco switches.
quux: Nope, managed. 45xx, 46xx, 69xx series
scratchme: Can't you configure it otherwise?
quux: I spent a lot of time on this about 2 years ago. And the answer was basically: nope
scratchme: Odd.
quux: You get the benefits of teaming when you have a lot of diverse hosts
quux: because basically, the switch decides which teamed nic to send to, based on the MAC address of the incoming packet
scratchme: Yeah, the auto addressing it does.
scratchme: Anyway, thx for your advice, but I need to sleep now.
scratchme: I really appreciate your insight on your setup.
scratchme: As I'm sure you know, that kind of insight doesn't come cheaply ;)
quux: so, (and this is fun!) another corrolary of that is, if all your hosts are talking to the server from one particular vlan, they all land on one of the teamed nics
quux: (because all come from one MAC: that of the vlan!)
quux: heh, it was a fun project. One I'd like to replicate into business environments
scratchme: Yeah, hadn't thought about that.
quux: Anyway, gnight. and Always a pleasure