Session Start: Sat Apr 07 17:07:46 2007
Session Ident: #bt-roundtable
|17:07:47| * Now talking in #bt-roundtable
|17:07:47| * zelazny.freenode.net sets mode: +ns
|17:08:39| * Joins: DWKnight (n=dwknight@hlfxns01bbg-142068089124.dhcp-dynamic.ns.aliant.net)
|17:14:33| * Joins: amc1 (n=Miranda@87.127.163.142)
|17:16:09| * Joins: uau (n=uau@dsl-hkigw7-fedddc00-236.dhcp.inet.fi)
|17:17:40| * Joins: robinp (n=rockinro@tlug/member/robinp)
|17:17:43| <@The_8472> hrrm... will everyone be awake&available the next few hours?
|17:17:48| * Looking up robinp user info...
|17:18:40| <@The_8472> alus said it's 7:00 in his timezone, so i guess he isn't awake yet
|17:18:49| <uau> i think 8
|17:19:12| <DWKnight> [12:19:02pm] [alus TIME reply]: Sat Apr 7 11:16:06 2007
|17:19:12| <DWKnight> -
|17:19:12| <DWKnight> [12:19:03pm] [alus TIME reply]: Sat Apr 07 08:19:15 2007
|17:19:23| <DWKnight> dunno if his script is having a fit or what
|17:19:33| <@The_8472> ^^
|17:20:18| <@The_8472> btw, who's robinp?
|17:21:04| <robinp> im just some random
|17:21:37| <robinp> i wrote the bittorrent-zeroconf protocol
|17:21:50| <@The_8472> hrmhrm
|17:22:13| <@The_8472> anyway, if everyone is still available in the next hours then we could just postpone it until shadow and alus are available
|17:22:14| <robinp> if this is private i will leave
|17:22:28| <robinp> im just interested
|17:22:55| <@The_8472> hrrm, as long as it doesn't get too crowded i don't have a problem
|17:23:10| <@The_8472> i just don't want noobs to interfere ;)
|17:23:32| <robinp> no - i was going to be ears only
|17:23:41| <uau> wasn't it TheSHAD0W who originally suggested this time?
|17:23:56| <@The_8472> i'd have to ask my external memory
|17:24:50| <@The_8472> yep, he did
|17:34:51| * Joins: piratepenguin (n=declan@212.2.183.41)
|17:48:05| * Looking up piratepenguin user info...
|19:01:04| * Joins: TheSHAD0W (i=theshado@66.18.161.41)
|19:01:08| <TheSHAD0W> Hi.
|19:01:12| <TheSHAD0W> Sorry I'm late.
|19:01:19| <TheSHAD0W> How much did I miss? :-P
|19:01:21| <@The_8472> you're not the only one :)
|19:01:23| <@The_8472> nothing
|19:01:31| <TheSHAD0W> Ah, good.
|19:01:41| <@The_8472> still waiting for alus
|19:01:55| * Quits: robinp (n=rockinro@tlug/member/robinp) ("Leaving")
|19:02:12| <TheSHAD0W> Were any other bt.com devs coming?
|19:02:30| <@The_8472> don't think so
|20:26:46| <TheSHAD0W> Well, I think we've gotten off to a smashing start!
|20:26:48| <TheSHAD0W> ^__^
|20:32:17| <@The_8472> yeah...
|20:32:50| <@The_8472> well, i guess we could start w/o alus
|20:32:57| <@The_8472> and fill him in later
|20:33:15| <TheSHAD0W> I x'pose so.
|20:34:10| <@The_8472> hrmhrm, where to start...
|20:34:40| <@The_8472> how about listing what we want to improve?
|20:34:48| <TheSHAD0W> Sounds good.
|20:34:58| <TheSHAD0W> (1) Tracker communication revamp.
|20:35:25| <TheSHAD0W> (2) Base protocol revamp (?)
|20:35:28| * Quits: piratepenguin (n=declan@212.2.183.41) ("Leaving")
|20:35:37| <TheSHAD0W> (3) Torrent structure revamp
|20:35:45| <@The_8472> ack, ack and ack
|20:35:52| <TheSHAD0W> (4) Torrent file format (bencode) revamp (?)
|20:36:11| <TheSHAD0W> Question marks mean they may already be good enough.
|20:36:37| <@The_8472> uau made good arguments against my proposal for 4, but maybe you have something better?
|20:36:56| <@The_8472> and i think the base protocol certainly needs a revamp
|20:36:57| <TheSHAD0W> Could you put your URL up again?
|20:37:10| <@The_8472> http://azureuswiki.com/index.php/User:The8472/Tech_Stuff
|20:37:21| <@The_8472> Enhanced Messaging & new messages
|20:37:22| <@The_8472> Tracker Auth
|20:37:27| <TheSHAD0W> When I say "base protocol" I basically mean the handshake and message format, not their content.
|20:37:27| <@The_8472> and Info dictionary 2.0
|20:37:45| <@The_8472> yes, i know... those need an overhaul too
|20:37:54| <@The_8472> to negotiate message IDs and stuff like that
|20:38:04| <@The_8472> the bitfield is too... implicit
|20:38:46| <TheSHAD0W> Well, let's take one thing at a time.
|20:38:57| <TheSHAD0W> Which do you want to argue over first?
|20:38:59| <TheSHAD0W> 8v)
|20:39:22| <@The_8472> how we should keep it with backwards compatibility
|20:39:36| <TheSHAD0W> I'd have to say drop backwards compatibility.
|20:40:10| <TheSHAD0W> Have a new extension for the new torrent files (xxx.bt_2 perhaps?) and a whole new format.
|20:40:11| <@The_8472> well, we must be able to cooperate with old/not-yet-compatible clients in some way
|20:40:21| <TheSHAD0W> Why?
|20:40:25| <TheSHAD0W> Okay.
|20:40:38| <TheSHAD0W> Let me make a statement...
|20:41:21| * @The_8472 busy-loops
|20:41:27| <TheSHAD0W> I'd rather not diverge from the original mission of BitTorrent, which means it is primarily a tool for CONTENT PRODUCERS. A producer puts up a web site, offers torrents and links to tools to download them.
|20:41:58| <TheSHAD0W> Obviously it can be used in different ways, but that was BitTorrent's primary mission and it's worked wonderfully that way.
|20:42:30| <TheSHAD0W> I think that a different P2P system for other purposes would be apropos, but I'd like to follow the primary mission with this particular spec.
|20:43:01| <@The_8472> yes, i have no problem with that, although it needs certain... adjustments to reality ;)
|20:43:27| <@The_8472> e.g. to cite myself: "The purpose of the new torrent file format is to produce only one info hash for identical content, regardless of variable factors like file names, file order or piece size."
|20:43:43| <@The_8472> to deal with the splitting of swarms
|20:44:06| <TheSHAD0W> Well, the reality is, torrents don't cooperate with each other, except for DHT-stuff.
|20:44:19| <@The_8472> hrm?
|20:44:23| <TheSHAD0W> So there's no real reason for a new-style torrent to cooperate with an old-style torrent.
|20:44:49| <TheSHAD0W> If someone wants to download a new-style torrent, all they need to do is upgrade their client, and then they have access to both. What could be simpler?
|20:45:28| <@The_8472> convincing people to use that
|20:45:49| <TheSHAD0W> Convincing them to upgrade?
|20:46:03| <@The_8472> to use v2 torrents
|20:46:13| * TheSHAD0W shrugs
|20:46:18| <TheSHAD0W> We can lead the horse to water...
|20:46:50| <TheSHAD0W> The fact that downloaders would be getting better performance would probably tend to influence the publishers.
|20:47:02| <@The_8472> hrrm, maybe.
|20:47:43| <@The_8472> my original idea was to make the tree buildable from the piece list... that's why i proposed a binary tree.
|20:48:19| <@The_8472> but then other problems would arise... i guess it wouldn't be compatible anyway
|20:48:26| <TheSHAD0W> Yup.
|20:48:42| <TheSHAD0W> And there's an inherent problem w/ keeping multiple hash formats in the same file...
|20:48:59| <TheSHAD0W> Which one do you use, and what happens if their values aren't compatible?
|20:49:11| <@The_8472> discard the entire torrent as invalid
|20:49:30| <TheSHAD0W> Yes, but there are problems evaluating that, especially ahead of time.
|20:49:49| <@The_8472> you can't do that ahead of time... but it wouldn't be worse than a fake torrent
|20:49:54| <TheSHAD0W> You'd probably wind up downloading, and sending out a bunch of bad packets, before you discovered the incompatibility.
|20:50:00| <TheSHAD0W> Eh.
|20:50:09| <TheSHAD0W> I would rather throw it all out and start anew.
|20:50:49| <TheSHAD0W> Anyway.
|20:51:15| <TheSHAD0W> We need to resolve this before we move on, it's very important.
|20:51:33| * TheSHAD0W pings everyone
|20:52:05| <@The_8472> wire protocol compatibility is not a problem, byte-pattern detection works fairly well, encryption has demonstrated that
|20:52:44| <@The_8472> although i would propose a compact meta-handshake to negotatiate the protocol of multiple are available to deal with that once and for all...
|20:53:17| <TheSHAD0W> Um.
|20:53:29| <TheSHAD0W> I really, REALLY have to vote against backwards compatibility.
|20:53:36| <@The_8472> like A: <magic constant><list of supported protoocols><i prefer protocol N> B: <magic constant><list of supported protocols><i choose M>
|20:53:44| <@The_8472> nono, not backwards
|20:53:46| <@The_8472> forwards
|20:54:07| <TheSHAD0W> Um.
|20:54:15| <TheSHAD0W> It's basically the same thing.
|20:54:58| <@The_8472> you would implement a new wire protocol exclusively? oO
|20:55:02| <TheSHAD0W> Yes.
|20:55:13| <@The_8472> w/o support for "classic" bittorrent?
|20:55:17| <TheSHAD0W> Correct.
|20:55:35| <@The_8472> well, i can tell you that others won't play along with that...
|20:55:35| <TheSHAD0W> The client could download on both old and new torrents, but there would be no "mixed" torrents.
|20:55:53| <@The_8472> i'm talking about the wire protocol here, not the .torrent format
|20:56:00| <TheSHAD0W> Yes.
|20:56:33| <TheSHAD0W> The client could use .torrent files and download using the classic BitTorrent protocol, or use the .bt_2 files and download on a whole different wire protocol.
|20:56:48| <@The_8472> ah... a all-in-one bundle, i see
|20:56:52| <TheSHAD0W> Yeah.
|20:56:56| <@The_8472> i'd treat those things seperately
|20:57:22| <@The_8472> because an "old" torrent still could benefit from the new wire protocol
|20:57:32| <TheSHAD0W> No, no, no...
|20:57:52| <TheSHAD0W> The new wire protocol wouldn't be used on old torrents.
|20:57:53| <@The_8472> there's no need to use the old protocol on old torrents when the new one performes better/supports more features
|20:57:57| <@The_8472> why?
|20:58:40| <@The_8472> µT implements pex for classic torrent, azurues has its own wire protocol for normal torrents, and still cooperates with classic clients w/o problems
|20:58:44| <@The_8472> i don't see the need
|20:58:48| <TheSHAD0W> Given the old torrent structure, I can't see any way to improve performance significantly by revamping the protocol.
|20:58:51| <@The_8472> esp. when compatibility is cheap
|20:58:57| <TheSHAD0W> Azureus has a different wire protocol??
|20:59:05| <@The_8472> yep
|20:59:12| <TheSHAD0W> Now that, I didn't know.
|20:59:15| <@The_8472> lol
|20:59:26| <TheSHAD0W> Got it documented anywhere?
|20:59:30| <@The_8472> nah
|20:59:36| <TheSHAD0W> X_X
|20:59:37| <TheSHAD0W> Geez.
|20:59:51| <TheSHAD0W> Anyway.
|21:00:08| <@The_8472> it negotiates all supported messages... works fairly well
|21:00:30| <@The_8472> this way we implement pex, chat, listening port announcement on outgoing connections etc. etc.
|21:02:05| <TheSHAD0W> ...
|21:02:54| <@The_8472> but really, there's no need to artificially limit backwards compatibility. If complicates the protocol too much to make it backwards compatible.... ok. But just restricting it to the new format is nuts
|21:03:16| <@The_8472> of course the new .torrent format will require new messaging, due to the hash-requests
|21:03:44| <@The_8472> but i can't see a reason why capable clients shouldn't benefit from PEX on old torrents
|21:04:33| <TheSHAD0W> Um.
|21:04:47| <TheSHAD0W> Your client already does PEX over something compatible with the old protocol.
|21:05:03| <@The_8472> no, not really
|21:05:09| <TheSHAD0W> No?
|21:05:16| <@The_8472> it's Az-specific, doesn't work with other clients
|21:05:24| <TheSHAD0W> Okay.
|21:05:30| <TheSHAD0W> Now I have a headache. :-P
|21:05:33| <@The_8472> µT has its own pex-messages based on the classic protocol (just consuming some message IDs)
|21:06:30| <TheSHAD0W> I think we're going to start getting into other arguments...
|21:06:45| <TheSHAD0W> I personally don't like PEX and think it may be harmful to swarms...
|21:06:55| <@The_8472> only if implemented carlessly
|21:07:13| <@The_8472> it actually merges swarms from different trackers and stuff like that
|21:07:18| <TheSHAD0W> And as for chat... I'm surprised it isn't being used to spam advertising...
|21:07:30| <@The_8472> it's not used at all :P
|21:07:35| <@The_8472> well... rarely
|21:07:35| <TheSHAD0W> LOL
|21:08:04| * TheSHAD0W should fix that, and make some money at the same time >;->~
|21:08:13| * @The_8472 pats TheSHAD0W
|21:08:31| <TheSHAD0W> Anyway...
|21:08:34| <@The_8472> the chat channels are torrent-specific, although virtual torrents can be created to open new "channels"
|21:08:38| <@The_8472> yeah?
|21:10:04| <TheSHAD0W> I think that Bram's intention to lock down the protocol has failed, and that we ought to add a flexible method for providing extensions... But I think connecting peers on an old-style torrent via a different wire protocol isn't necessary, and if it requires complicating the wire protocol to do it, harmful.
|21:10:06| <TheSHAD0W> KISS.
|21:11:09| <@The_8472> "and if it requires complicating the wire protocol to do it, harmful." <- it doesn't
|21:11:25| <@The_8472> well, not much
|21:11:26| <TheSHAD0W> Hum.
|21:11:32| <TheSHAD0W> Heh.
|21:11:38| <TheSHAD0W> I think it does.
|21:11:43| <TheSHAD0W> If only the handshake.
|21:11:43| <@The_8472> we'd need a flag in pex messages like "crypto, new, classic"
|21:12:15| <TheSHAD0W> So you want to include PEX as a basic part of the protocol.
|21:12:36| <@The_8472> the protocol itself should be message-purpose-agnostic
|21:12:40| <TheSHAD0W> Not sure how well that'll be received by the mainline folks.
|21:13:08| <@The_8472> my idea is that named and versioned message-"packages" consuming a chunk of the message ID space are negotiated
|21:13:28| <@The_8472> and PEX messages would be part of one of those packages
|21:14:15| <@The_8472> this way you can have a BT-baseline package that essentially mirrors the old bittorrent message, then v2 (that's the versioning part) of BT-baseline with fast extensions
|21:14:34| <@The_8472> and either a v3 for pex, hash requests etc. or a seperate package BT-extras
|21:15:16| <@The_8472> this way other clients could freely implement client-specific packages like chat or whatever comes to their mind w/o risking incompatibility
|21:19:42| * @The_8472 stares into the silent shadow...
|21:20:23| <TheSHAD0W> I think it'd be a lot simpler to keep the two protocols separate.
|21:20:40| <TheSHAD0W> The client contacts a classic-style tracker, gets an IP, and initiates a classic-style connection.
|21:20:50| <@The_8472> errm...
|21:21:05| <TheSHAD0W> It doesn't have to try and negotiate with the remote peer what sort of connection, whether the remote peer is old or new...
|21:21:51| <@The_8472> you're thinking too impractical... if a new .torrent needs new-protocol-capable peers (which is almost inevitable) AND new trackers it's almost impossible to get support for that.
|21:21:56| <TheSHAD0W> If the new protocol is indeed superior, after a year or so there won't BE any old .torrents.
|21:22:10| <TheSHAD0W> So there's no sense complicating the new protocol in order to account for it.
|21:22:26| <@The_8472> but if you can use the new messaging protocol on old torrents you'll get more support for the messaging... which means in turn it'll be easier to get support for new torrents
|21:22:35| <TheSHAD0W> ...
|21:22:46| <TheSHAD0W> No, if anything it'd extend the life of the old format.
|21:22:57| <@The_8472> and there's no need for a new tracker protocol just to support the new .torrents, because they still would rely on an infohash, it just has another meaning
|21:23:03| <TheSHAD0W> The clients would be able to handle it just fine w/o trying to mix protocols.
|21:23:26| <TheSHAD0W> Well, there are other reasons why I'd want to revamp the tracker protocol.
|21:23:35| <TheSHAD0W> But that's actually separate.
|21:23:41| <@The_8472> a modular approach means that things can be implemented step by step instead as one huge ball of... well... incompatible stuff
|21:23:51| <uau> back
|21:23:59| <TheSHAD0W> WB
|21:24:09| <@The_8472> uau, tell TheSHAD0W that he's wrong :P
|21:24:50| <TheSHAD0W> Whether the new client spec is compatible w/ the old tracker spec is actually beside the point.
|21:25:25| <TheSHAD0W> You can make them compatible w/o mixing wire protocols.
|21:26:25| <TheSHAD0W> I just don't want to take a brand spanking new protocol and then warp it in order to accomodate old clients and torrents.
|21:26:26| <@The_8472> the point is that you only need minimal protocol-awareness in your peer sources (DHT, PEX, tracker) to get an almost collision-free support for mixed swarms
|21:26:46| <TheSHAD0W> Urk.
|21:26:56| <@The_8472> nah i don't want that either. the necessary changes are minimal
|21:28:22| <@The_8472> incoming connections can be easily distinguished by whatever magic constant you use in the wire protocol, so only outgoing connections have to pick the right protocol...
|21:29:45| <TheSHAD0W> "Magic constant"?
|21:30:04| <@The_8472> e.g. \19bittorrent protocol
|21:30:39| <@The_8472> and then \xxnew fancy protocol name
|21:30:55| <@The_8472> and if neither matches... assume crypto
|21:31:08| <@The_8472> it's trivial
|21:31:19| <TheSHAD0W> So... What do you send when you're initiating a connection?
|21:31:48| <@The_8472> <@The_8472> the point is that you only need minimal protocol-awareness in your peer sources (DHT, PEX, tracker) to get an almost collision-free support for mixed swarms
|21:32:01| <@The_8472> if pex tells you that peer is new-protocol-capable you use the new protocol
|21:32:13| <TheSHAD0W> And what about the tracker?
|21:32:32| <TheSHAD0W> An old-style torrent is probably going to specify an old-style tracker.
|21:32:53| <TheSHAD0W> Which means it's going to hand you peer data w/ no details about whether those peers are compatible w/ old/new protocol.
|21:32:53| <@The_8472> well, then you have to upgrade the link after the fact, like azureus does
|21:33:07| <TheSHAD0W> Yuck.
|21:33:11| <@The_8472> it works
|21:33:21| <TheSHAD0W> I'm sure it does.
|21:33:22| <TheSHAD0W> But it's messy.
|21:33:26| <TheSHAD0W> KISS.
|21:33:27| <@The_8472> and it's only a transitional measure that can be phased out at some point
|21:33:51| <@The_8472> it is not neccesasry, but it makes transition easier
|21:34:04| <@The_8472> and makes the protocol more attractive to implement
|21:34:08| <TheSHAD0W> I still have to disagree.
|21:34:17| <TheSHAD0W> It makes it more ANNOYING to implement.
|21:34:26| <@The_8472> you don't HAVE to implement it...
|21:34:38| <TheSHAD0W> You can leave the old torrents on the old protocol and they'll just work.
|21:34:55| <uau> if you're talking about wire protocol then i don't really see a need for any kind of compatibility to previous protocol
|21:35:05| <@The_8472> sure, but you wouldn't have the new features (mainly pex+listening port annoucement)
|21:35:09| <uau> "old-protocol" clients are free to use the new protocol as their private extensions
|21:35:19| <uau> but that doesn't need to concert the protocol standard itself
|21:35:32| <uau> concern
|21:35:32| <@The_8472> uau, i'm trying to tell that to TheSHAD0W...
|21:35:39| <TheSHAD0W> It means old torrents won't have those features, but they'll still work.
|21:35:59| <uau> The_8472: you seem to be saying that "compatibility" should be in the standard itself
|21:36:01| <@The_8472> but he insists on limiting the new protocol to new .torrents... i.e. disallowing mixed swarms
|21:36:24| <@The_8472> uau, not in the protocol standard, only in some things based on it (e.g. pex)
|21:36:28| <uau> the protocol can be specified like that
|21:36:56| <uau> if azureus wants to implement a private extension which matches the new protocol and use it for old-style torrents it's of course free to do that
|21:37:15| <@The_8472> i think we just should standardize that "private extension"
|21:37:30| <@The_8472> as a transitional advisory thingy
|21:37:41| <@The_8472> that's all
|21:38:06| <@The_8472> that all clients could implement that "private extension" in the same way
|21:39:03| <TheSHAD0W> Well, you'll be free to publish the extension to allow other devs to implement it.
|21:39:11| <@The_8472> ...
|21:39:31| <@The_8472> i want everything in ONE specification, including _optional_ backwards compatibility extensions
|21:39:48| * TheSHAD0W has a headache
|21:40:17| <@The_8472> you'll get used to it, it took us 4-5 of such talks just to develop the crypto stuff
|21:40:24| <@The_8472> this here will take longer...
|21:41:35| <@The_8472> anyway, it's as simple as saying if bit XY in the classic bittorrent handshake means that you must upgrade to the new messaging protocol as soon as the handshake is over... it doesn't need alterations in the new protocol... it just a few extra lines in the spec under the chapter "compatibility" or whatever...
|21:42:19| <TheSHAD0W> ...
|21:42:23| <TheSHAD0W> Can you do me a favor?
|21:42:30| <@The_8472> depends on the favor
|21:42:45| <TheSHAD0W> Can we work on an optimal new wire protocol, and THEN consider what's necessary to run old-style torrents over it?
|21:43:00| <@The_8472> yeah, why not
|21:43:42| <uau> related to crypto stuff, hopefully this time there won't be anyone to say "hey we agreed on the overall design an hour ago already, the byte-level protocol must be frozen now so we can make a release"...
|21:44:08| <TheSHAD0W> LOL
|21:44:12| <@The_8472> uau, no pressure this time ;)
|21:45:00| <TheSHAD0W> Okay. Resolved: Backwards compatibility is optional, and the goal is to produce the best wire protocol possible before considering backwards compatibility. Agreed?
|21:45:09| <@The_8472> ack
|21:45:20| <@The_8472> although minimal changes should be acceptable
|21:45:33| <TheSHAD0W> Yes.
|21:45:47| <TheSHAD0W> Minimal as defined as, not needing 500 lines of code to implement. :-P
|21:45:53| <@The_8472> ^^
|21:46:10| <@The_8472> gzip your code :P
|21:46:16| <TheSHAD0W> Step 2: Nail down the data structures for the new-style torrent.
|21:46:34| <@The_8472> as in... replacement for bencoding yes/no?
|21:46:36| <TheSHAD0W> The hash structure, etc.
|21:46:40| <TheSHAD0W> No.
|21:46:41| <@The_8472> ah, ok
|21:46:49| <TheSHAD0W> That comes later.
|21:46:57| <@The_8472> http://azureuswiki.com/index.php/User:The8472/Tech_Stuff#Info_dictionary_2.0 <-
|21:47:11| <TheSHAD0W> In a word: Yuck.
|21:47:13| <TheSHAD0W> 8v)~
|21:47:24| <@The_8472> looks more complicated than it is
|21:48:07| * TheSHAD0W studies
|21:48:53| <@The_8472> although fail-fast behavior is paramount when parsing that dictionary...
|21:49:09| <TheSHAD0W> "fail-fast behavior"?
|21:49:37| <@The_8472> throw an error as soon as you discover the slightest deviation from the spec
|21:49:48| <TheSHAD0W> That's obvious.
|21:49:52| <TheSHAD0W> Um.
|21:50:01| <TheSHAD0W> You seem to be missing filenames. 8v)~
|21:50:10| <@The_8472> tell that to people who don't sort their dictionaries...
|21:50:19| <@The_8472> nope, i don't :)
|21:50:38| <TheSHAD0W> * <N> keys (byte arrays): <sha1 file hashes> => dictionaries:
|21:50:39| <TheSHAD0W> * key (string): "length" => integer value: file length in bytes
|21:50:39| <TheSHAD0W> * key (string, optional): "tree" => byte array: <file tree root key>
|21:50:46| <@The_8472> * key (string): "fileMap" => list of <file mapping type>
|21:50:46| <@The_8472> * entries => dictionary (is <file mapping type>)
|21:50:46| <@The_8472> * keys (string) <file/dirname> => integer <file indices> || dictionary <file mapping type>
|21:50:57| <TheSHAD0W> Okay.
|21:51:02| <TheSHAD0W> I wouldn't separate those.
|21:51:15| <@The_8472> the file/directory structure is seperated from the immutable information
|21:51:18| <TheSHAD0W> Put 'em in a single array.
|21:51:21| <uau> The_8472: are there non-sorted dictionaries somewhere in practice?
|21:51:26| <@The_8472> uau, yes...
|21:51:28| <uau> i haven't seen errors related to those
|21:51:36| <DWKnight> improperly sorted dictionaries at least
|21:51:52| <@The_8472> azureus reparses the .torrent and sorts the dictionaries in the process
|21:52:07| <@The_8472> which results in a different infohash, people recently "complained" about that...
|21:52:16| <uau> i think the client i usually use would show an error/warning from those
|21:52:25| <uau> and i don't ever remember seeing one
|21:52:48| <DWKnight> some .net (vb.net/c#) based torrent makers had problems
|21:52:49| <TheSHAD0W> BitTornado will die given a dict that's out of order - I think.
|21:52:55| <DWKnight> mainly relating to the .utf-8 crap
|21:53:16| <uau> so exactly what client would such a torrent work in ???
|21:53:21| <uau> utorrent?
|21:53:22| <@The_8472> µT
|21:53:29| <@The_8472> and mainline, i think
|21:53:48| <TheSHAD0W> Anyway...
|21:53:53| <@The_8472> yeah, back2topic
|21:54:09| <TheSHAD0W> Before we even work on this, I'd like to go to a more basic level.
|21:54:15| <TheSHAD0W> And that's the hash structure for the torrent.
|21:54:16| <uau> not in earlier mainline versions anyway, i haven't checked latest
|21:54:31| <TheSHAD0W> Proposal - that each file in a torrent have its own root hash.
|21:54:56| <@The_8472> that makes hash-requests... messy
|21:54:57| <DWKnight> turn multi-file torrents into compound single-file torrents kind of idea?
|21:55:31| <TheSHAD0W> A bit. But it also makes the file structure neater, makes it easier to download individual files...
|21:55:43| <uau> TheSHAD0W: i think it shouldn't be a _root_ hash
|21:55:51| <uau> at least not necessarily
|21:55:53| <TheSHAD0W> ?
|21:56:03| <uau> if the torrent has lots of files
|21:56:06| <@The_8472> btw, my proposal already includes optional per-file trees
|21:56:13| <@The_8472> for exactly that purpose
|21:56:18| <TheSHAD0W> It's also not that difficult to map it out and assign a linear number for each hash.
|21:56:43| <uau> The_8472: i think the way your proposal did that was clumsy
|21:56:55| <uau> as it meant separate out-of-alignment pieces over the same file
|21:57:05| <TheSHAD0W> When you start doing optional stuff it gets messy. KISS.
|21:57:25| <@The_8472> if you say KISS one more time i'll start eating kittens!
|21:57:31| <TheSHAD0W> KISS.
|21:57:34| <TheSHAD0W> >;->~
|21:57:36| <@The_8472> yummy... kittens :]
|21:58:07| <TheSHAD0W> I was actually talking this over w/ Bram a while back.
|21:58:21| <@The_8472> *munch* anyway... *burp* ... piece alignment might be a good idea, but that either means that torrents become a loose bundle of files or the tree building becomes more complicated
|21:58:30| <TheSHAD0W> He thought giving each file its own root hash could make .torrent files unmanageably large...
|21:58:49| <TheSHAD0W> But since each file would also need its own name, a multi-file torrent would be large anyway.
|21:58:58| <DWKnight> multi-file torrents aren't exactly clean as things stand right now
|21:59:06| <TheSHAD0W> DWKnight: Boy howdy.
|21:59:20| <uau> you don't necessarily need to make the per-file hashes root-level ones and include them in the .torrent
|21:59:21| <@The_8472> well, my proposal aims at cleaning that mess up, a bit at least
|21:59:37| <TheSHAD0W> uau: No, but it makes things simpler.
|21:59:41| <@The_8472> but if someone has a better idea, i'm all ears
|21:59:54| <uau> i think that pieces should be aligned to the start of each file
|22:00:09| <TheSHAD0W> Mmmmmmm...
|22:00:20| <TheSHAD0W> That IS a good idea.
|22:00:52| <uau> i'm not 100% sure whether they also need to be separate tree per file, or whether you could just form a single tree and the alignment would just mean some pieces are shorter
|22:00:57| <@The_8472> although it makes implementations a bit more complex, because you'll have more than one small piece
|22:01:11| <TheSHAD0W> I think it's a great idea.
|22:01:19| <TheSHAD0W> It wouldn't make things that much more complex.
|22:01:39| <TheSHAD0W> You could fix it in an abstraction layer between files and pieces.
|22:01:39| <@The_8472> a single tree certainly would be a good idea, as long as you can rebuild a single-file-tree from it whenever you have to
|22:02:14| <@The_8472> i mean as long as you have all leafs
|22:02:16| <TheSHAD0W> Okay.
|22:02:26| <TheSHAD0W> Before we figure this out, let's go even lower...
|22:02:34| <TheSHAD0W> Is there any reason to switch from SHA1?
|22:03:19| <@The_8472> sha1 might be broken in the "near" future, but there is no widespread replacement atm
|22:03:41| <@The_8472> and everyone else sticks to it too, even high-security applications... no need to bother
|22:03:45| <@The_8472> yet...
|22:03:45| <uau> sha-256 is a kind of replacement and reasonably widespread AFAIK
|22:03:58| <uau> but it does take more resources
|22:04:19| <@The_8472> cryptographers feel uneasy with sha256 because it has a very similar structure, so breaks in sha1 could lead to breaks in sha256
|22:04:47| <@The_8472> but we're talking years here
|22:05:05| <TheSHAD0W> SHA-256 is 32-byte, correct?
|22:05:34| <@The_8472> yep
|22:05:44| <@The_8472> the raw output that is...
|22:06:03| <TheSHAD0W> Heh.
|22:06:04| <TheSHAD0W> Uh oh.
|22:06:17| <@The_8472> of course you could just shorten it to 160 bits
|22:06:36| <TheSHAD0W> Mmmm, n/m
|22:06:39| <TheSHAD0W> Yeah.
|22:07:19| <@The_8472> anyway, i don't see a need to switch hashes now... i've only included the sha1 key too keep the door open for future changes
|22:07:35| <TheSHAD0W> Okay.
|22:08:12| <TheSHAD0W> So the current plan is (1) to use SHA1, and (2) to align files to piece boundaries.
|22:08:18| <TheSHAD0W> All in favor?
|22:08:41| <DWKnight> aye to 2 for sure
|22:08:41| <uau> ok, though 2) needs more detail
|22:08:41| <@The_8472> ack; ack pieces to file boundaries
|22:09:12| <TheSHAD0W> uau: Yup.
|22:09:32| <TheSHAD0W> Okay. I am now going to make another proposal.
|22:09:37| <@The_8472> i still think a binary tree would be the most flexible structure
|22:10:07| <TheSHAD0W> Um.
|22:10:14| <TheSHAD0W> I've been thinking about it...
|22:10:16| <@The_8472> this way could address everything in the tree structure...
|22:10:28| <@The_8472> like request: tree depth 17, node 40
|22:10:31| <TheSHAD0W> And it really makes selecting what you want to download next a huge bear.
|22:10:48| <uau> one problem with the most trivial implementation of (2) is that if we talk about the client-protocol level (HAVE) pieces then you don't want to increase those a lot if there are small files
|22:10:49| <TheSHAD0W> Yeah, it makes requests easy.
|22:11:14| <@The_8472> haves could have variable resolution too
|22:11:24| <TheSHAD0W> uau: I don't think that's going to be a huge problem if we bunch up have messages.
|22:11:34| <@The_8472> have: depth 0, node 1 would mean you have everything
|22:11:46| <TheSHAD0W> I mean, ten thousand small files, if you bunch the messages, is a 10K block.
|22:12:19| <uau> with what kind of "bunching" protocol?
|22:12:22| <@The_8472> why so complicated? we don't have to announce haves at the 16KiB leaf level
|22:12:23| <TheSHAD0W> The_8472: BTW, with a proper tree you can serialize nodes and just use a single number.
|22:12:32| <TheSHAD0W> uau: Queueing them and then sending them together.
|22:12:42| <uau> you'd need to optimize it somehow to only use a byte per piece
|22:12:44| <@The_8472> a stacked have message
|22:12:48| <TheSHAD0W> Oh, sorry. 40K.
|22:12:55| <TheSHAD0W> For 10,000 files.
|22:13:30| <@The_8472> that's too expensive for the initial bitfield or whatever has to replace it
|22:13:50| <TheSHAD0W> Hm.
|22:14:11| <@The_8472> but really... it doesn't make any sense to have haves at the leaf level... because distinct files only stick out at the leaf level of the tree
|22:14:29| <TheSHAD0W> I don't think it'll come up often and it won't be too harmful if it does.
|22:15:39| <@The_8472> leaf level is 16KiB or less (if the files are smaller than that)
|22:15:49| <TheSHAD0W> Yes.
|22:15:55| <TheSHAD0W> 16K sounds good.
|22:16:32| <TheSHAD0W> Piece level is configurable, but will probably be 256K. Piece level is the minimum size for a have message.
|22:16:45| <TheSHAD0W> Might send have messages in the form of a tree above that level...
|22:17:27| <@The_8472> btw... we could just say that only files above a certain threshold are aligned
|22:17:41| <TheSHAD0W> Yeah, but again, that complicates things.
|22:17:43| <@The_8472> i mean it's pointless to align some 1kb "comment" files...
|22:17:55| <TheSHAD0W> How do you handle them if they aren't aligned?
|22:18:09| <@The_8472> just as we would now
|22:18:15| <TheSHAD0W> An add-on to the piece on the end of another file? A separate list?
|22:18:16| <TheSHAD0W> Eh.
|22:18:41| <@The_8472> hash over a block of unaligned files instead of a file-by-file basis
|22:18:43| <TheSHAD0W> The problem is, it winds up that you need that little comment file in order to verify the other file.
|22:19:02| <@The_8472> only if the other file is below that threshold too
|22:19:20| <TheSHAD0W> I think it'd just be easier to piece-align everything, no matter what size. You know what I'm gonna say next. 8v)
|22:19:41| <TheSHAD0W> KISS! (Hope you have room for 2nds.)
|22:19:45| <@The_8472> that can be exprensive if you have tons of smaller-than-16KiB-files
|22:20:22| <@The_8472> now i'll just fry them...
|22:20:27| <TheSHAD0W> Yeah... But (1) those'll probably be archived together, and (2) that's already expensive in torrent size...
|22:20:37| <@The_8472> hrrm, ok
|22:21:15| <TheSHAD0W> IMO if you want a nice hardy system it has to be simple.
|22:21:24| <TheSHAD0W> Even if that adds inefficiencies.
|22:21:59| <TheSHAD0W> Blah. AFK a min or two.
|22:24:19| <@The_8472> i've been thinking... alignment only works if you get the leaf hashes, i.e. for all 16KiB blocks. Only then it's possible to take all hashes of the leafs and transform them into a single-file-tree for "seperate" downloading. That's ineffective because you'd have to transfer all leaf hashes... not just some upper-level hashes, because higher level hashes are probably unaligned to tree you'd want to build
|22:24:40| <@The_8472> so including the per-file-root is still necessary, even if you align the pieces in the overall-tree
|22:25:42| <uau> depends on whether you align at the 16 KiB level or at the HAVE level (i'm not sure which one the earlier discussion was about)
|22:26:59| <@The_8472> it wouldn't matter
|22:27:36| <uau> you wouldn't need ALL leaf hashes covering the file anyway
|22:28:20| <@The_8472> well, only those haven't downloaded yet
|22:29:14| <uau> if you have a binary hash tree, then at most you'd need to get 2*tree_height extra hashes total
|22:30:24| <uau> when going up a level in the tree you need at most one "outside" hash on the "before" side and one on the "after" side
|22:30:37| <@The_8472> if the file-tree is unaligned to the global tree you'd need all leafs of the level that is aligned among both trees
|22:31:05| <uau> no you'd need 2*tree_height extra hashes total at most
|22:31:10| <@The_8472> no... if they're not aligned then higher levels are completely incompatible
|22:31:19| <uau> compared to the situation where you'd have a single-file torrent for that one file
|22:31:35| <@The_8472> think about it... alignment errors lead to totally different hashes on all levels except the base
|22:32:29| <uau> yes the hashes change, but that doesn't mean you'd need to get more than 2*tree_height extra
|22:32:38| <uau> compared to what you'd need to get anyway to download it as a single-file torrent
|22:32:52| <@The_8472> i think we're talking about different things
|22:33:30| <uau> then what are you talking about?
|22:33:34| <@The_8472> i'm talking about the scenario where i have the global tree and then want to construct a single-file tree
|22:33:55| <uau> i'm talking about how much "extra" hash downloading you have to do to pick a single file out of a batch torrent
|22:33:58| <@The_8472> or rather... compare if they're identical
|22:34:02| <uau> compared to downloading a single-file torrent for that file
|22:34:35| <TheSHAD0W> There's two ways to handle it.
|22:34:55| <@The_8472> uau, if you just download a single file in a batch-torrent swarm you're right
|22:35:03| <TheSHAD0W> (1) is to have the binary tree go all the way down to the leaf hashes, and send verification information as requested for each leaf.
|22:35:12| <@The_8472> but i was talking about the scenario where you construct the single-file tree to search for a single-file-swarm
|22:35:18| <uau> The_8472: so you're talking about such a scenerio, but you didn't specify what you're measuring in that scenario
|22:35:54| <TheSHAD0W> (2) (my recommendation) is to verify on the piece level, and when you find a verification error, then download the leaf hashes and replace them as necessary.
|22:36:08| <@The_8472> you can have 1 and 2
|22:36:22| <TheSHAD0W> Yes, but (2) requires less overhead.
|22:36:32| <TheSHAD0W> Especially since errors shouldn't be common.
|22:36:50| <@The_8472> if you have a binary tree you can just request a higher hash, and then construct the tree up to that hash upon verification
|22:37:22| <TheSHAD0W> No, it goes the other way.
|22:37:26| <@The_8472> but that means hash checking involves the construction of a mini-tree
|22:37:33| <uau> The_8472: but when it doesn't match you want to know which lower-level hash was incorrect
|22:37:52| <TheSHAD0W> If you use (2) you don't need to keep the binary tree abstraction to the bottom level.
|22:37:53| <@The_8472> uau, of course... if that happens you still can request leafs
|22:38:04| <TheSHAD0W> The piece hash can simply be a hash of the list of leaf hashes.
|22:38:46| <@The_8472> tree to the bottom allows you to do flexible have, request etc. addressing
|22:39:11| <TheSHAD0W> If you're that flexible it causes other problems.
|22:40:15| <@The_8472> hrm, yeah... keeping track of what the peer has might be messy... you'd have to implement a very compact tree data structure
|22:40:31| <TheSHAD0W> Yes, and searching for what's the rarest data it has gets REALLY messy.
|22:41:10| <@The_8472> not THAT messy...
|22:41:17| <@The_8472> you'd just need a global-availability tree
|22:41:27| <TheSHAD0W> X_X
|22:41:29| <@The_8472> but yeah, you have a point there
|22:42:37| <@The_8472> ok, so haves should be kept at a list or range format
|22:42:39| <TheSHAD0W> If the piece size is small enough, it really won't help speed start-up if you generate abstraction at a lower level.
|22:42:55| <TheSHAD0W> That's what I recommend.
|22:43:41| <TheSHAD0W> I also recommend that each have checksum be a hash of the simple list of leaf checksums, rather than a tree.
|22:44:14| <@The_8472> i'm not 100% sure if that's ok
|22:45:01| <TheSHAD0W> It could be done as a tree, just to make it neater, but I think it'd be added overhead for no reason.
|22:45:16| <TheSHAD0W> Let's say piece size is 256K, leaf size is 16K.
|22:45:25| <@The_8472> it wouldn't add overhead... only CPU usage
|22:45:33| <TheSHAD0W> Yes.
|22:45:47| <TheSHAD0W> It'd use a little RAM, but whatever.
|22:45:49| <TheSHAD0W> Thing is...
|22:45:52| <TheSHAD0W> Let's say piece size is 256K, leaf size is 16K.
|22:45:56| <@The_8472> yep
|22:46:05| <TheSHAD0W> That means a list of leaves per piece is 160 bytes.
|22:46:25| <TheSHAD0W> You download a full piece, verify it, and it's bad.
|22:46:37| <@The_8472> then you can request the leaf hashes
|22:46:45| <TheSHAD0W> So all you need is to obtain the 160 bytes from another peer to verify at leaf level.
|22:46:55| <@The_8472> yep
|22:47:43| <@The_8472> btw, hash-requests should really operate on tree-based addressing
|22:48:35| <@The_8472> because you want to verify only small chunks of the file during startup and haven't built your hash list yet
|22:49:32| <@The_8472> with higher-level hashes you can chop off large branches of the tree and focus on the lower-level hashes you currently need
|22:50:20| <@The_8472> ahh... another problem... where do we get hashes from?
|22:50:48| <@The_8472> i guess we could just request hashes from those peers that have the necessary pieces.. but getting high-level hashes might be hard
|22:53:38| * Looking up TheSHAD0W user info...
|22:54:13| * Disconnected
|22:54:57| * Attempting to rejoin channel #bt-roundtable
|22:55:25| * Rejoined channel #bt-roundtable
|22:55:33| <TheSHAD0W> You'd do that while downloading, but that'd add overhead to all your transactions.
|22:55:49| <The_8472> what was my last line?
|22:56:10| <TheSHAD0W> Whereas if you grab it as a list after the fact you only do so when you have bad data, which should be a rare event.
|22:56:20| <TheSHAD0W> <The_8472> then you can request the leaf hashes
|22:56:55| <The_8472> http://pastebin.ca/428811
|22:58:04| <The_8472> ah, hi alus
|22:58:10| <uau> throttling of hash requests is an issue in the client protocol
|22:58:21| <uau> you might reasonably want to get large amounts of hashes
|22:58:29| <uau> when checking a partially-correct existing file
|22:58:38| <The_8472> well, only allow larger amounds when the peer is unchoked
|22:58:48| <The_8472> *amounts
|23:00:01| <TheSHAD0W> Um.
|23:00:02| <The_8472> hrrm... or just allow hash requests _only_ when unchoked
|23:00:23| <The_8472> i mean for the usual mode of operation you could just request-hash, request-piece
|23:01:08| <The_8472> ah... no... you have to verify the hashes first...
|23:01:13| <TheSHAD0W> You don't want to retrieve the entire tree route for each piece. It gets very inefficient.
|23:01:20| <The_8472> so when the torrent starts you'll need a larger setof hashes
|23:01:31| <TheSHAD0W> I kinda recommend retrieving the entire hash list before downloading.
|23:01:48| <TheSHAD0W> Rather than trying to portion it out.
|23:01:53| <uau> what do you mean by "entire"?
|23:01:59| <TheSHAD0W> At the piece resolution.
|23:02:02| <The_8472> he means the have-level
|23:02:24| <The_8472> but that can be huge
|23:02:30| <uau> get the hash of every piece at HAVE level as a start-up phase before starting real download?
|23:03:00| <TheSHAD0W> 40 gig torrent, 256K piece size, 160,000 pieces, 3.2 megabytes. Yucch.
|23:03:01| <The_8472> we're talking about 256k pieces here
|23:03:08| <The_8472> yep...
|23:03:57| <The_8472> so you'd rather want to walk down the tree to individual pieces until you have enough peers that are interested in you and unchoke you so you can request the whole list
|23:04:11| <TheSHAD0W> Well.
|23:04:26| <TheSHAD0W> (1) given a tree, you can incrementally download the data.
|23:04:52| <TheSHAD0W> (2) if you're verifying a torrent, you're almost certainly going to need the whole list, especially if you're not allocating the files in full at the start.
|23:05:17| <TheSHAD0W> I think we're going to need to develop something as a hybrid.
|23:05:34| <The_8472> <TheSHAD0W> (2) if you're verifying a torrent, you're almost certainly going to need the whole list, especially if you're not allocating the files in full at the start. <- at that point you should already have it
|23:05:53| <TheSHAD0W> Mmm.
|23:05:54| <The_8472> things should be preserved across sessions ;)
|23:06:01| <TheSHAD0W> Assuming something doesn't corrupt your data...
|23:06:07| <TheSHAD0W> Which happens too often. :-P
|23:06:33| <The_8472> the tree should be stored in the same location where you store your download descriptions
|23:06:34| <TheSHAD0W> I'm thinking maybe it should be a tree, but not binary.
|23:06:43| <TheSHAD0W> Yeah...
|23:06:43| <The_8472> and if that gets corrupted you're in trouble anyway ^^
|23:07:40| * TheSHAD0W thinks
|23:08:15| <uau> i think non-binary cannot give any major advantages (it can be somewhat more efficient but i doubt it'd be a major factor)
|23:08:19| <The_8472> of course you could just fetch the hash from 2 peers and trust them if they're identical
|23:08:50| <The_8472> during startup that is...
|23:09:19| <TheSHAD0W> Well.
|23:09:34| <TheSHAD0W> What you would do is fill the tree.
|23:09:39| <The_8472> but that's up to the implementation, more or less... we should just provide a default practice
|23:10:18| <TheSHAD0W> You download the inverse path for a specified level just before you start downloading the piece.
|23:10:33| <TheSHAD0W> So binary tree? Yes?
|23:10:37| <The_8472> yep
|23:10:42| <TheSHAD0W> Okay.
|23:10:58| <The_8472> now that i'm thinking about it
|23:11:05| <TheSHAD0W> Let's choose.
|23:11:23| <TheSHAD0W> A binary tree down to the leaf level, or a binary tree down to the piece level, and then lists to the leaf level.
|23:11:35| <TheSHAD0W> The first is simpler. Adds a bit of CPU overhead...
|23:11:36| <The_8472> if a peer A has a piece 1, he should already have all necesasry hashes to build the tree up to its root... unless he has a lazy approach
|23:11:55| <TheSHAD0W> If he's announcing a have for it, he'd better not be lazy. :-P
|23:12:19| <The_8472> or the good-faith approach... whatever
|23:12:25| <TheSHAD0W> Heh.
|23:12:26| <The_8472> <The_8472> of course you could just fetch the hash from 2 peers and trust them if they're identical <-
|23:12:40| <TheSHAD0W> Rather not do that.
|23:12:51| <TheSHAD0W> You could download a hell of a lot before you figured out someone was lying to you.
|23:12:54| <The_8472> the problem is: how do we decide which hashes a peer has?
|23:13:18| <TheSHAD0W> If a peer announces it has a piece, it should have the hashes for it. Period.
|23:13:36| <TheSHAD0W> It might even have to calculate them dynamically; but it should have them.
|23:13:57| <The_8472> iff nobody is lazy and always posses the necessary hashes to verify a piece we can infer that he'll has all hashes for the pieces he has announced PLUS the required hashes to build a minimized tree
|23:14:13| <TheSHAD0W> Yes.
|23:14:37| <The_8472> if we do assume that they might be lazy we only know that they'll have the hashes for the pieces they annouce via haves
|23:15:00| <TheSHAD0W> If they're being lazy, they shouldn't announce availability.
|23:15:03| <The_8472> would a seperate hash-bitfield make sense?
|23:15:08| <TheSHAD0W> You should only send a have if you've verified the info.
|23:15:14| <TheSHAD0W> Er?
|23:15:26| <The_8472> ok, i take that as a no :)
|23:15:33| <The_8472> then we'll use auto-inference
|23:15:40| <TheSHAD0W> ?
|23:15:52| <The_8472> deducing which hashes a peer must have from his haves
|23:16:43| <TheSHAD0W> You can deduce that he has the hashes for all nodes above that level, and for all nodes opposing that level.
|23:16:54| <TheSHAD0W> Er, all nodes opposing those nodes.
|23:17:03| <The_8472> yes
|23:17:45| <The_8472> so, essentially... we'll have to construct a minimal hash-availability tree for every peer from his piece-haves
|23:17:48| <The_8472> fun fun fun...
|23:18:02| <TheSHAD0W> Naw.
|23:18:19| <TheSHAD0W> You don't have to construct anything until you want to request hash data from him.
|23:18:47| <TheSHAD0W> You keep a piece list for each peer, just like current BT.
|23:18:53| <TheSHAD0W> Sort 'em like current BT.
|23:18:57| <The_8472> well...
|23:19:12| <TheSHAD0W> And then when you're ready to request a piece, THEN you figure out what hashes you need from him.
|23:19:19| <The_8472> if you want to check which peer can offer you the necesary hashes then you'll have to keep such a tree in memory
|23:19:24| <TheSHAD0W> Oh, you're talking about verifying local data?
|23:19:31| <The_8472> or do you want to construct them all the time?
|23:19:52| <The_8472> local data about remote peers
|23:19:55| <TheSHAD0W> I would simply request the needed hash data from a peer that unchokes you before downloading piece data.
|23:20:05| <TheSHAD0W> But again we're getting off track.
|23:20:15| <TheSHAD0W> We need to pin down the structure of the hash data first.
|23:20:36| <The_8472> binary tree or tree down to a certain level and then a list is not such a huge difference
|23:20:47| <uau> i'm not sure whether it's better to try to decide structure of the data or typical peer interaction first
|23:20:48| <TheSHAD0W> No...
|23:20:58| <TheSHAD0W> Hm.
|23:21:08| <TheSHAD0W> uau: I don't see how it makes a difference really.
|23:21:18| <The_8472> it only makes a difference if we're talking about transforming multifile-trees into single-file trees
|23:21:24| <uau> requesting hash data from peers before you start downloading the piece also has a problem with slow peers
|23:21:36| <TheSHAD0W> uau: I know. :(
|23:21:49| <uau> if you only requests the hash from one of them you could end up in a situation where you get the piece but didn't get the hashes to verify it
|23:21:55| <uau> before all the peers with that piece choked you
|23:22:11| <TheSHAD0W> uau: Then you'd just have to wait 'til another peer w/ that piece unchoked you.
|23:22:19| <TheSHAD0W> And then that'd be the first thing you request.
|23:22:36| <The_8472> or you request hashes first... and then the pieces
|23:22:40| <TheSHAD0W> You MIGHT wind up with a situation where you'd get repeatedly zapped by slow peers...
|23:22:50| <uau> The_8472: that doesn't work
|23:22:57| <TheSHAD0W> Actually, it might.
|23:23:08| <uau> unless you request the same hash multiple times, for every peer you download from
|23:23:24| <TheSHAD0W> The hash data is fairly small; you could request it from each peer you download that piece from until you get it, then stop.
|23:24:02| <TheSHAD0W> You'd wind up definitely having the hash data before getting any piece data that way.
|23:24:14| <TheSHAD0W> Question...
|23:24:17| <The_8472> nah, that's too much wasted data
|23:24:24| <The_8472> yes?
|23:24:32| <TheSHAD0W> 40 gig torrent, 256K piece size, what's the maximum number of hashes you'd need to retrieve?
|23:24:55| <TheSHAD0W> 64 gig torrent, say.
|23:24:59| <TheSHAD0W> Same # of hashes.
|23:25:12| <The_8472> worst-worst case, dumb implementation case?
|23:25:12| <TheSHAD0W> 1024*4*6.
|23:25:15| <TheSHAD0W> Er, no.
|23:25:19| <TheSHAD0W> 10+4+6.
|23:25:27| <TheSHAD0W> 20 hashes, 400 bytes.
|23:25:30| <The_8472> ah, for a single piece
|23:25:53| <The_8472> btw, you'll end up "filling" the top of the tree pretty fast
|23:26:06| <TheSHAD0W> Yes, and you'd reduce the request size similarly.
|23:26:14| <The_8472> so any further requests will get smaller and smaller
|23:26:14| <TheSHAD0W> So the average would be more like 100 bytes.
|23:26:45| <TheSHAD0W> And I don't think requesting that in duplicate would be that awful, especially since you usually wouldn't START downloading that same piece from multiple peers most of the time.
|23:27:46| <TheSHAD0W> Can we nail down which tree format we want? Please? 8v)
|23:28:00| <TheSHAD0W> Binary all the way down, or hybrid?
|23:28:07| <The_8472> that depends on the alignment stuff
|23:28:25| <TheSHAD0W> Alignment would still be at the piece level.
|23:28:25| <The_8472> we introduced the alignment stuff to allow easy transitions between multi and single file torrents
|23:28:26| <uau> TheSHAD0W: i think nothing gives a big advantage over binary
|23:28:32| <TheSHAD0W> At the have level.
|23:28:46| <The_8472> if we use alignment at the have level you could use a list
|23:28:48| <uau> TheSHAD0W: so i think it's at least not a major/urgent decision
|23:29:03| <TheSHAD0W> Yes. But I'd like to pin it down.
|23:29:23| <TheSHAD0W> List from leaf to have level, or binary all the way down?
|23:29:37| <TheSHAD0W> Binary is slightly more overhead... But it's consistent. Simpler.
|23:30:02| <uau> The_8472: you mentioned trying to find a single-file torrent based on a multifile one; that was not a use case for which i introduced it
|23:30:15| <The_8472> TheSHAD0W mentioned it
|23:30:23| <The_8472> uau, anyway... what WAS the case?
|23:30:47| <The_8472> to allow you to synchronize 2 torrents?
|23:30:58| <uau> resuming partial single-file downloads from a multifile one
|23:31:09| <The_8472> ?
|23:31:26| <uau> resuming a single complete file as part of a multifile torrent without overwriting parts of it or tricky special case handling
|23:31:26| <The_8472> you mean a seperate single-file torrent, right?
|23:31:33| <The_8472> ah
|23:32:00| * Joins: Nolar (n=Nolar@azureus/Nolar)
|23:32:10| <The_8472> i think that's not so important, most clients already handle that stuff properly by simply storing the remaining chunks of the piece somewhere else
|23:32:13| <uau> with the current format the first doesn't work even if the piece size is the same because alignment differs
|23:32:43| <The_8472> it only works on single-file torrents, because different piece sizes are multiples of each other
|23:33:08| <The_8472> but really, if we do the alignment stuff just for selective downloading we could just drop it
|23:33:18| <uau> "handle that stuff properly"?
|23:33:38| <The_8472> properly == w/o redownloading
|23:33:46| <uau> you mean clients would automatically detect that a single file is (mostly?) complete and decide not to store data inside it?
|23:34:01| <uau> does some client really do that?
|23:34:05| <The_8472> hrm?
|23:34:13| <The_8472> ok, now i don't quite get what you mean
|23:34:40| <uau> "resuming a single complete file as part of a multifile torrent"
|23:34:53| <uau> so you already have a file which exactly matches a file from the multifile torrent
|23:34:59| <The_8472> yes
|23:35:05| <uau> however because the first and last pieces also overlap other files they're not complete
|23:35:23| <uau> thus most likely a client will reload those pieces completely
|23:35:39| <uau> and while doing that overwrite the existing file and possibly corrupt it
|23:35:58| <The_8472> why would that corrupt it if the file matches?
|23:36:17| <The_8472> if the whole file except the edge pieces match the edge pieces should match too
|23:36:32| <The_8472> can't see how that would corrupt the file
|23:36:35| <uau> it does match the ideal contents of the batch torrent
|23:37:00| <uau> but that would only mean it doesn't get corrupted if the client only writes data that has already been hash checked
|23:37:09| <uau> and keep everything else in memory or temporary files
|23:37:19| <The_8472> huh?
|23:37:32| <uau> and also doesn't store any other data out-of-place over not-downloaded pieces
|23:37:39| <The_8472> ok... i have a complete file from a single-file-torrent, right?
|23:38:16| <The_8472> and now i want to integrate that single file into a multifile torrent that should have identical content
|23:38:36| <The_8472> how exactly would that corrupt the single-file-thingy?
|23:38:44| <The_8472> or require any special treatment
|23:38:53| <The_8472> either they are identical or they arent
|23:39:00| <uau> assume everything you download is all zeros (the real contents of the torrent are not)
|23:39:20| <The_8472> k
|23:39:44| <uau> thus if you store anything in the actual files instead of temporary files or memory before hash checking the files will be corrupted
|23:40:04| <The_8472> so i'll ban the peer sending corrupt data and then redownload it
|23:40:26| <uau> and the file will be corrupted until you do (which might be forever)
|23:40:29| <The_8472> you're saying that someone is sending corrupt data, i can't see how that's a problem of a multifile torrent
|23:40:53| <TheSHAD0W> I have to leave.
|23:41:13| <The_8472> cu
|23:41:27| <TheSHAD0W> Have fun. :-)
|23:41:30| * Quits: TheSHAD0W (i=theshado@66.18.161.41)
|23:41:32| <uau> a complete file which matches a file in the batch torrent shouldn't get corrupted
|23:41:51| <The_8472> WHY should it get corrupted?
|23:41:56| <The_8472> i don't see the point here...
|23:41:59| <uau> i just explained that?
|23:42:08| <The_8472> no, you said someone is sending corrupt data
|23:42:17| <The_8472> that's a problem that has to me eleminated, sure
|23:42:25| <The_8472> but that's not a problem specific to multifile torrents
|23:43:34| <The_8472> i can't see why we sould align pieces just to work around someone integrating a singlefile torrent into a multifile torrent at the same time when someone else sends corrupt data
|23:43:49| <The_8472> that's a _really_ odd scenario
|23:44:38| <The_8472> doing the piece alignment stuff to allow you to seek a singlefile torrent for a file within a multifile torrent makes somewhat sense if you want to merge all sources you can get... but that has additional requirements...
|23:44:52| <uau> the scenario is simply already having a complete file from a multifile torrent
|23:45:05| <uau> which isn't that odd
|23:45:13| <The_8472> if it's complete it can't get corrupted, ever
|23:45:31| <uau> with what assumptions?
|23:45:36| <uau> with current clients it certainly can
|23:46:08| <The_8472> i don't know how other clients do that... but azureus stores the entire edge pieces to ensure that hash checks never fail
|23:46:25| <The_8472> so once it finished it stays that way
|23:47:11| <uau> sounds like you're talking about something else (finishing enough of the torrent to have a complete file)
|23:47:27| <The_8472> complete file + edge pieces, yes
|23:47:41| <The_8472> <uau> the scenario is simply already having a complete file from a multifile torrent <-
|23:47:43| <uau> the assumption was just that you already have a complete file
|23:47:57| <The_8472> so... from an external source?
|23:48:07| <uau> from whatever earlier source
|23:48:25| <The_8472> and only that file, no neighboring files?
|23:49:55| * Quits: uau (n=uau@dsl-hkigw7-fedddc00-236.dhcp.inet.fi) (Read error: 131 (Connection reset by peer))
|23:52:28| * Joins: uau (n=uau@dsl-hkigw7-fedddc00-236.dhcp.inet.fi)
|23:52:31| <The_8472> ok, so the process is: i download file A. then i download a multifile torrent, which includes A. then i integrate A into the not-started multifiletorrent. Then i check it and it'll miss the edge pieces. Then i'll attempt to download the edge pieces and someone is sending fake data.
|23:52:35| <The_8472> right?
|23:52:57| <uau> yes
|23:53:09| <The_8472> and you want to workaround THAT?
|23:53:30| <uau> or if you use incremental allocation you write other pieces over the edge pieces
|23:54:02| <The_8472> that's more of an implementation problem than something that should be fixed by the protocol imo
|23:54:08| <uau> or if you specifically mark pieces you're starting to download on disk you also corrupt the file
|23:55:22| <The_8472> you could just copy the file before you attempt such "risky" things (after all you must be assuming that someone is hellbent on sending you corrupt data), you could store partial pieces in seperate files/memory....
|23:55:22| <uau> working around that problem is not the ONLY motivation
|23:55:46| <The_8472> well... if you have more realistic scenarios... please share them
|23:55:52| <The_8472> but that's a pretty weak one
|23:56:08| <uau> i think that's realistic enough
|23:56:18| <The_8472> nah
|23:56:30| <uau> for example you could simply write protect the file
|23:56:42| <The_8472> even better :D
|23:56:50| <The_8472> anyway, it's a corner case at best
|23:56:58| <uau> but then that will fail even if nobody tries to send you bad data
|23:57:10| <uau> if the client tries to redownload the edge pieces
|23:57:18| <uau> unless it has special-case handling
|23:57:22| <The_8472> really, how often do you get corrupted data?
|23:57:48| <The_8472> the whole scenario relies on that...
|23:58:21| <uau> i don't think so
|23:58:54| <The_8472> having singlefile torrent (likely) + having a matching multifile torrent (not day-to-day use, but some people do that from time to time) + someone sending corrupt data on exactly those pieces combined with the previous requirements (very unlikely)
|23:59:00| <uau> i think the scenario is realistic enough if you simply have a complete file already and want to use it as part of the batch resume
|23:59:04| <uau> without risking corruption
|23:59:27| <The_8472> then implement the store-in-memory-before dumping-to-disk stuff...
|23:59:45| <uau> you don't need to assume corruption actually happens in this particular resume to qualify it as a use case
|23:59:58| <uau> that would be quite inefficient
Session Time: Sun Apr 08 00:00:00 2007
|00:00:04| <The_8472> then use temporary files
|00:00:14| <uau> also inefficient
|00:00:28| <The_8472> whatever... but that scenario doesn't justify rearranging the hash trees
|00:00:32| <uau> and anyway that's not the only use case
|00:00:41| <uau> there's also resuming a partial download
|00:01:13| <The_8472> which would include finding a matching multi-to-single file torrent
|00:01:28| <The_8472> if that's our use case we'll have to do more
|00:02:07| <uau> what do you mean by "matching multi-to-single file torrent"? that sounds like a weird term
|00:02:18| <uau> "multi-to-single-file"?
|00:02:45| <The_8472> err
|00:03:09| <The_8472> finding a matching singlefile torrent for a file from a multifile torrent
|00:04:05| <uau> or the other way around
|00:04:12| <The_8472> yes
|00:04:13| <uau> you download part of a single-file torrent
|00:04:32| <uau> then the site combines recent single-file torrents to a batch torrent
|00:04:33| <The_8472> anyway, it would involve verifying that your tree is part of the other tree
|00:04:46| <uau> and the single-file torrents either disappear or the batch torrent is much faster
|00:07:09| <uau> you wouldn't need to check any special tree-inside-tree containment
|00:07:35| <uau> if you have a generic routine to resume partial downloads you can use that
|00:08:16| <The_8472> well, you can't be sure that the file actually matches the torrent until you have checked the pieces and rebuilt the tree under the new torrent, so it would be better if you can transform the old tree to the new one
|00:08:26| <The_8472> but that only works if the trees are aligned, not just the pieces
|00:08:38| <uau> of course you could avoid downloading lowlevel hashes for the resume if you recognize that you already have some of those hashes from a previous torrent
|00:09:16| <uau> the trees do NOT have to be aligned (or them not being aligned would not cost much at least)
|00:10:06| <uau> if you have a binary tree you can "realign" it by knowing one extra hash per level at both boundries
|00:10:08| <uau> boundaries
|00:10:33| <The_8472> only for the base level
|00:10:39| <The_8472> any higher levels can't be realigned
|00:11:55| <uau> why do you think they couldn't?
|00:12:42| <The_8472> draw a tree with 16 leafs, and then attempt to create a subtree of 8 leafs, starting with the second leaf. you will se that you can't construct the 2nd tree when you don't have the leafs, because the nodes closer to the root will mismatch
|00:13:24| <uau> but you can have the leafs
|00:13:41| <The_8472> if you have the leafs it's as expensive as getting the whole list
|00:14:09| <uau> the leafs are the same if you align them
|00:14:14| <The_8472> well, almost... assuming that the pieces are randomly distributed
|00:14:28| <uau> so having them doesn't require you to ask for them in the "new" torrent
|00:14:29| <The_8472> 50% completion, randomly distributed pieces
|00:15:36| <The_8472> anyway, alignment does make resuming files on a different torrent easier, but we shouldn't make things too complex
|00:16:07| <The_8472> it's important to retrieve the hashes from other peers efficiently, with minimal overhead
|00:16:12| <The_8472> esp. on slow torrents
|00:22:15| <The_8472> alus, btw... you haven't voiced yourself yet...
|00:22:19| <The_8472> *poke*
|00:23:11| <alus> voiced?
|00:23:33| <alus> I'm still not sure what you guys are talking about
|00:23:57| <uau> the most recent part was mainly about aligning pieces to file start in batch torrents
|00:24:00| <The_8472> heh, ok... we're talking about the hash tree stuff in the new .torrent format