Hello everyone - I am a software developer who enjoys working on p2p technologies. I see great potential in what you folks are doing. P2P will do wonders for distributing large and/or popular files over the internet. I'd like to find out more about how Red Swoosh works. I hope you guys can help me in answering a few of these questions...
How does file integrity verification/hashing work? Does the RedSwoosh client verify downloaded files against some hash? If so, how does it know the hash of each file and/or file block in advance, since the original content lies on an HTTP server?
Do the RedSwoosh servers store/cache any files themselves?
Are there any mechanisms in place to ensure the fair sharing of resources? How does RedSwoosh combat freeloaders? (e.g. BitTorrent has this whole choking algorithm thing)
What about privacy? Can't anyone easily determine what IP addresses recently downloaded file X, just by sending a request to the Red Swoosh servers? Does the user really have to manually delete files via the control panel each time?
How can the user turn Red Swoosh off and/or pause traffic, without completely uninstalling it?
How large can the Red Swoosh file store/cache get?
Are there any plans to open source Red Swoosh? What about making the protocol open? Will Red Swoosh be releasing any technical documentation anytime soon? I'd love to develop some applications to work on top of Red Swoosh...and I think it would seriously benefit your cause.
How many users does Red Swoosh currently have? How many users are online at any given time?
Yes, that's a lot of questions. I greatly appreciate your help and insight. I wish you all the best of luck in the p2p caching business!!!! Take care!
Hi vptes11, thanks for the great questions. Let’s take those one at a time.
1) The first time a user clicks on a swooshed link, our servers go out and download it straight from the webserver. We record some HTTP headers, its full-file SHA1 hash, and a series of 256KB block hashes that span the entire file. All this is recorded by our servers for future reference. When the next user clicks on that swooshed link, we tell the client all this information, and the client uses this to verify the data as its downloading from the webserver and other peers. Thus we detect and correct natural corruption that happens over the internet (rare, but increasingly probable the larger the file), and any intentional corruption that a hacker could theoretically inject.
2) No, Swoosh hosts no content whatsoever. We only record “metadata” on the files, such as hash, URL, filesize, HTTP headers, etc.
3a) We believe Swoosh makes the fairest and best use of resources by only sharing unused bandwidth. What that means is we’ll measure how much bandwidth you appear to have available, measure how much you are currently using, and only consume a fraction of the remaining. If you start some download (via us or any other application) or fire up a VoIP call or whatever, we’ll detect that and immediately ramp down our usage. Clearly, this is a huge topic, and I’d welcome any other questions you have on it.
3b) As for "freeloading", it doesn't really happen because Swoosh is a "caching" architecture versus a "download" architecture. In the latter, everyone is miserly with bandwidth because once the download finishes, it's totally forgotten -- if the client hasn't "paid" the network for the download by the time it completes, it never will. But in a caching network such as ours, you're still a member of the network after the download completes. Thus you don't need to worry about sharing when you're doing the download itself or even immediately thereafter -- we'll wait until you go to bed or are using Word, and share quietly in the background. We make the "cost" of sharing so negligible that freeloading isn't worth it. Truly -- if you notice Swoosh at all, it's a bug, and we intend to fix it. In a caching network such as ours, there's a vast *overabundance* of seeds, so the problems inherent in an artificially bandwidth-scarce downloader network simply don't exist.
4a) First, you’re right, users can always delete the files they download, either via our control panel, or via an interface supplied by the website where they downloaded it. Furthermore, predelivered files naturally expire when their TTLs or expiration dates run out, on-demand files expire after 40 days (or as necessary to free up disk space for other on demand downloads), and so on. Content management is a tricky art, and I’d welcome any suggestions you have on how to improve in this area.
4b) As for the privacy implications, a rogue client could in theory learn what other IP addresses have a given file – this is no different than any other P2P network. However, unlike other networks, we can and do strictly regulate who can join the network in the first place, and thus make every effort to keep rogues out.
5) Our next release has a full client-side control panel (ie, shows up right next to your other control panels) that allows you to see and manage all content you’ve downloaded from all websites, pause predeliveries from a given website, or go offline entirely. The intent is to give you full awareness and control over the client’s activity; if you’re interested please write me at [email protected] and I’ll add you to our early-beta list. We’ve love to get your input.
6) We’re still trying to figure out the right balance. Clearly, if a user has disk space and asks us to download a file, we want to do that. But we also don’t want to fill up his hard drive. So right now we use a rule where we’ll download a file if there is twice as much space on the disk remaining. It’s not a great rule, and if you have any suggestions on how to improve it, we’re all ears.
7) We’ve strongly considered the open source approach, and are trying to figure out the right way to get there. Perhaps we can review the merits of this in another discussion.
8) My quick queries tell me it’s seen 2851194 unique users to date, and 569377 unique files. But both of those numbers seem low; I’ll need to check with my guys for more accurate numbers.
Thanks for all your questions, and we’d love to hear more.
Hey David - Thanks for your answers. I greatly appreciate the time you've taken to help me understand a little of what's going on "under the hood". I definetly see a bright future for Red Swoosh, and p2p in general.
In point #3, you assume that freeloading won't be a problem. I contend that it will be: 1) Given the huge privacy implications (everyone can see what you've downloaded), people will want to run hacked versions of Red Swoosh, versions that do not cache and serve files. 2) People do not want their scare upstream bandwidth to be filled up, even if RedSwoosh promises to be nice. ISPs do not want this either. They may ban and/or block RedSwoosh clients that aren't freeloaders. 3) People do not want cached files filling up their hard drive space. Again, they'll prefer to run a hacked version of Red Swoosh that doesn't cache anything.
You may think that most people aren't saavy enough to reverse engineer your protocol and write a hacked client, but once someone writes one, it will be availible on the net for free. And it will become popular in a matter of hours. Think KaZaA and KaZaA Lite.
Also, how does your download algorithm work? Meaning, I assume that it downloads files sequentially. But how does it manage downloading a file from multiple peers? Does the algorithm pick and choose favorites among the list of peers? Does it measure the speed of each peer? How do your servers choose what peers go on the list? How long is the list? Etc.
Regarding the freeloader question, I won't pretend it's an easy topic. And I'm not going to claim there won't be people who will try it. I merely claim the incentive to do so is very low.
- As to your privacy point, "everyone can see what you've downloaded" is a bit misleading. In actuality, nobody can see what you've downloaded except for you, us, and the website you downloaded it from. Indeed, it's little different from content downloaded direct from a webserver.
- As for the scarcity of upstream bandwidth, I entirely agree people want to conserve it. But due to our caching architecture, clients share only when it's most convenient, and bandwidth is least scarce. Thus rather than magnify traffic spikes (by requiring that you share at the same time while receiving), we tend to smooth them out (by sharing when the network is idle). This is friendly to individuals, friendly to organizations, and friendly to ISPs.
- And as for the point about caches filling up, our client never downloads anything you didn't ask it to, and you are always able to delete whatever it has. And even if you don't delete it, it'll naturally expire.
Clearly we're entering the subjective realm of what motivates the hacker mind, and there are as many justifications as there are hackers. But at the end of the day our system is very fair and strives to maintain the highest standards of security, privacy protection, efficiency, and performance.
Again, we're not trying to make bold claims in a vacuum. Please give it a shot and try it out. If you feel that it is not living up to the standards we've sent, please let us know and we'll fix it. We don't claim to have perfected the formula. But we have a great product that people love. Give it a shot and you might find that you love it too.
netlama -- Sorry for the slow response. As Travis mentioned, we're in the midst of a total rewrite of the downloader, and the new method doesn't require using UPnP to reconfigure NAT mappings. Once complete, the new method will leave your existing UPnP mappings unchanged.
Yes, basically it starts with the standard STUN / simultaneous-connect NAT-traversal algorithms and then adds some secret sauce on top of that. The theories on how to make it work are all pretty well known, but the implementation holds some interesting challenges.