Brainstorm
 

Hello,

I was sitting here while downloading programs on a PcLinux system. At current with the 70kbs download it's gonna take 1.5 hours. I came up with an idea while sitting here that if I knew howto program I would release a linux distro and use my idea. Now the idea:

Why not have when you install and option to donate like 2kbs upload to help the community? Like you get 5000 users of this distro who are interested in this, thats 10,000kbs upload speed to help the main servers. This could be something like torrents(?) but updates. I don't really know how it would work because I don't know howto program. The people I told about this thought it would be great.

I see alot of people coming on here looking for a project, here's one.

I've gotta say, I really like this idea, too!
It's almost identical to a P2P network (like torrents, Limewire, KaZaA, etc.), but the only things that would be a cause for concern, is that getting the distro widely distributed enough to make the update sharing worthwhile is a pro-bono marketing task on its own.

Aside from that, it's still an amazing idea.

EDIT: Instead of creating an entirely new distro, why not assemble a small programmers group, and see if you can present an alpha-version of a product to present to an already widely-distributed distro (or several). :D

I'm not sure what you mean, but it sounds exactly like BitTorrent to me.

Might work like that don't really know. This idea is for updates. Like lets use debian for example. You do an apt-get install ???, apt looks at the server site for references to people who have ??? file. site says these people at this ip have it, apt turnes around and downloades the file via what the debian tells it. The server does help with the download but the end-users that help contribute do as well. Kinda got a philosophy like bittorent but not really. This would be done for updates and software installs. This would make end-users seeders in a sorts for the updates. Like I said 2kb is a small amount to donate and would give great amounts of upload speeds the server never could give.

I'll put it in perspective:

You install this GNU/Linux. I am also interested in this distro so I download the .iso. After I install the distro I decide to update all the software.So I plug in the command to update. My machine contacts update server and server reports you have the updates I require, so my machine recieves the info needed to locate you and download from you and the server at the same time. Now put this in bigger perspective server has 5000 people who have the files I need but the server is swamped today because this distro just announced a whole bunch of security updates. So this server can serve me up a whooping 35kbs of download rate. My idea would also use the 2kbs the 5000 people are sharing to get the files I need easly increasing my download rates to max my isp can handle. Like I said thats 10,000kbs upload rate! Thats a very awsome number and would greatly reduce stress on servers.

It does sound like bittorrent, but as far as I know Bittorrent is only used at the moment for distributing ISOs and illegal movies.

I like the idea of bittorent being used somehow for downloading updates. Basically, once you have installed the updates then you can start sharing the updated packages. Instead of sources.list (for Debian) pointing to an official mirror, it could point to a torrent tracker, that tells the package manager what updates are required and which torrent to get them from. (something like this anyway)

--Ian

Bittorrent is not as good for updates as donkey - there is built-in search in Donkey and no need to explicitly think about everything - just put to sharde dir. BT is for ISO's. You've got me on your side - I've shared my distributed-in-source folder..

Wow, someone even downloaded bzip2, and grub has 142% upratio. A bit strange - I don't understand 1 request and >100% UpRatio, but nevertheless the idea is viable here and now. Everybody using any P2P, share all fresh packages you have!

I think it sounds cool, and I reckon the thread title would be a kickass name for it, too.

Ynot Irucrem:
I completely agree. :D

i get the whole 'sounds like bittorrent' thing, but i dont get whether or not you want a package manager that uses means similar to bittorrent to download files, or if you just want a stand alone client that downloads things, a la bittorrent...?

btw: if youre aiming towards the bittorrenty package manager, libtorrent is a good place to start (to anyone looking at this as a project). if i was working on my own package manager, i'd definately take this idea on...

My newbie 2 cents:

Bittorrent is only fast when you have a lot of piers, right? And perhaps not that fast when u get to small files. Maybe u need an update manager/package manager that will try ftp if bittorent is to slow. Or another distributed p2p that works with small files? Then u need some kind of checksum check for each file? right? maybe u can get these of the main repositry. Maybe u can use Par2 that also can repair files.

here's the thing though: i linux community would undoubtedly yeild a significant amount of peers (forced seeding might have to implemented on the manager). plus, there could be seed servers, whose sole purpose is to seed packages. i get where your coming though. an alternative within the manager would definately be needed. as per checksums, that could be done/downloaded when the repo listing are updated.

I think that's why mirrors exist though - to help spread out the load on the main server and placing mirrors geographically closer to its audience while maintaining the integrity of the files being downloaded.

Bittorrent is great, but in reality I find it much slower than using mirrors that are closer to me PLUS I feel guilty about using somebody else's personal bandwidth without being able to contribute much with my own slow connection. And obviously being in a region of the world with slow internet connectivity, I personally haven't experienced any speed advantages of torrents over mirrors either because the bottleneck is at my ISP's end. Bittorrents are very slow where I live... because I probably have to connect to a guy who's living half a world away, whereas I'm much better off connecting to a Singapore or Korean mirror.

well, ideally there could be seeding mirrors at location. plus, youd be getting seeders near you aswell. ofcourse, an idea like this might not benefit everyone. dialup users arent going to see that huge increase in speed that others might, but its a great idea none the less for people who want to speed up mass package distribution. mirrors get slow when alot of people want to use them, torrents get faster...

Maybe it could be good alternative to use distrubuted p2p for "servicepacks" - a compilation of all updates since a distro release up to say last month for example. That would work for anyone updating a brand new install. And then the update program would find the updates necesary since that update.

And maybe only parts that are popular downloads would be included in files shared via torrent. The rest might as well run of the server, right?

The more and more the discussion of this idea develops, the more and more I like it and see a VERY easy way to do it.

Making use of libtorrent is almost a must (or translating the needed libtorrent functions into another language - like Lisp, which is what I would be writing this in - if there is no cross-language bridge API, but I don't see that being a problem).

As for forced seeding? Not a problem either, just have a daemon running on startup that acts as a torrent seeder, update checker, etc.

I definitely see this as a worthwhile project. I'm seriously considering stopping a few of my current projects, and on my days off (from work), starting development on this idea; I'll use any excuse to further my Lisp skills.

My current idea on the program structure would be:
Layer 1 - The daemon(s) for seeding, update checking (which would be completely optional and customizable), the torrent libraries (like transfer protocols and algorithms).
Layer 2 - The functions and procedures responsible for storing the updates and patches, and applying them to the corresponding application(s).
Layer 3 - The user interface (GUI- and CLI-based configuration interfaces).

why lisp?

im really into this idea too. i wouldnt stop my main project, but id definately be willing to pitch in. i think something like this would warrant the creation of a distro too. just create a new .deb or .rpm distro that uses this manager. if this gets going i want in.

edit:

so as to not be completely counter productive, in questioning lisp, i'd suggest c++ or python for this project.

Python would probably be the best bet, as I'm guessing that libtorrent is written in Python. I dropped the Lisp-bomb because ever since I started learning it, and I've found all of my development times to be pretty much cut in half. Plus, because I haven't been programming in Lisp for too long for it to become a natural mental flow (not to mention that the syntax logic for Lisp is COMPLETELY backwards from any other language), I don't want to confuse myself with learning Python. I probably will learn Python anyways, as I've been meaning to for a while.

It's great that you're interested, and I would love to start this as a formal project. I'll try and set up a section for this on my website (which is still in the "setting up" stage), but I really don't want this to be an idea that just gets thrown out there and is found collected electronic dust in a few months.

:D
I'll definitely get on making this something official, because, like I've said, it's definitely a damn worthwhile project.

libtorrent is written for c++, but there is a python wrapper. python just seems like it would be simpler.

if you want you could just set up a sourceforge project for this. theyll give you web space and such. keep me informed.

cheers to the OP!

I don't think a new distro would be the greatest idea, it would be easier to just do the program itself and distribute packages or even tarballs. Setting up a new distro entails:
1. more work
2. bigger download
3. testers have to install a distro just to test it.

Otherwise, sounds good.

i think a distro is warranted, but not full fledged new one. just implementing this on an .rpm based system would be ideal. even something like modifying gentoo's portage to use this to download large files would be awesome.

Wow...You guys sure came up with some good ideas to add. I would like to ask..If someone took up this project, would I be able to somehow be involved? I spit the idea out and would love if someone could give me some involvement in the project. If I knew how to program I would jump right on this myself. Maybe an idea committee I could belong to? Of course I would donate the bandwidth to help the project grow. Maybe I could help with a wiki or something? Who knows.

theres no reason you can't be a part of it. if youre interested in programming, this is an opportunity to learn some stuff. if you aren't into that sort of thing, you could contribute with documentation and such. you could even start the project page for us.

Details
 

Ok..The submission for the acceptance into sourceforge has been submitted. I named the project Pkgswarm and just have to wait for approval from sourceforge for the project to be started. If I can somehow learn to program from this it would be great. I'll pitch in in every way I can and try to learn to do some programming. I will update this post to let all know the status of my submission. I licensed it under the GPL.
.

Whichever language you learn, corbintechboy, I'll be in the same boat as you.

The only languages I ever took the time to learn, were BASIC (and dialects), Java (*regurgitation sound*), and Lisp; nothing like C or C++.

ive been looking around and it looks like python-libtorrent is depreciated by flood. i think python would be the best way to go about making this a reality.

i like that name btw :D

I like the name too.
But what happened to the thread? I can hear the crickets chirping.

did sourceforge ever confirm the project? we should start emailing each other and probably find another dev or two...

I'd be interested in helping w/ this dev project. I've got minimal C/C++ experience, but I can learn. I'll check out the sourceforge site later.

absolutely. my email's nicksaika<at>gmail<dot>com

furthermore, what's the language we will be utilizing, C++ or Python?

with regards to python of c++, it really doesnt matter. python is really good stuff, and it might be the easiest way to implement something like this. i have some ideas that i want to discuss with you about what the application will _actually_ do. i figure there will be a few parts to the project, so itll take a while to get it done.

the sourceforge page still isnt up, but it should be up soon. hopefully the OP will post in here once its up. once thats up correspondence will be even easier.

my email should be in my profile for this site. if its not just make a post and ill put it up. microsoft/linux, i doubt this will require _that_ much experience, and either way its a good learning experience.

at this point, im thinking python might be best for this project. ill tell you why once we get more into corresponding and discuss what we want pkgswarm to really be.

Still waiting on a response! I will post when I do get it. Man I want to get this started so I can learn to program! Sounds fun.

nick, i dropped you an email. ms/lin and corbin, you guys may want to email him to or email me (dforsythe<at>gmail<dot>com) so that i can get your email addresses. anyone else interested in this project feel free to email me too.

Ha1f, I Sent you and email as well. Is there any news on the sf.net site?

the sf.net proposal is still under review. i sent you an email back looking for ideas.

Could we get it hosted on Google code to start off with, just to get the ball rolling? Base it on the Bittorrent 4.x python code so we don't reinvent the wheel so much?
From 4.4 onward I think, Bittorrent started needing a lot of extras to work, I'm guessing people would look for a light package manager?

Also, change the name to swarmp: SoftWare Automatically Revamped for Many People (I was bored :p)

we already have a sourceforge page. http://sourceforge.net/search/?type_...words=pkgswarm

check it out and email one of us if you want to get involved.

sawwwback: i got your message, but you blocked the email feature for your own account. email me at the address i posted in this thread. we appreciate the help :-)

Apologies about that, just emailed you :-)

the way i understand it each of the existing mirrors for a repo would act as both a tracker and a perminant seeder


mabie you could have seeding active whilever the pakage update daemon is running like PUP allways runs in the background in FC and zen in Suse

I don't have that much free time, but I could probably stand to do some less involved things like bug hunting, docs, and the SF website.

I would really like to see this as not an entirely new package manager, but just a layer on top of apt/yum/etc, maybe with a plugin architecture, as there will always be more package managers, and this would allow other people to write plugins for their own. This might mean a lot of initial work before anything usable is produced, seeing as everyone in this thread is on different distros/BSDs (I read your profile, Ha1f :) ), but people are going to be less wary of picking up a layer on top of their favourite package manager (with all it's features) than an entirely new PM that they might not consider as good, and more popular and well-known the project is, the more people will join in, and the more chance it will have of surviving. Writing a new package manager will also be a lot of work, you should be able to just stand on the shoulders of the existing ones, and if need be, this part can be written later anyway.

I really like the idea of this project and it has the potential to be something big, and I don't want to see it just die. A lot of projects haven't gone into the design phase enough, and just stagnate when the coder/s comes up against a brick wall. A lot of discussion and planning is still needed before anyone even thinks of writing code, so that will give people a chance to research everything involved (Python, daemons, bittorrent, package managers..). Python is a great first language, so you (us) coding newbs can help too.

Disclaimer: As you can read from my sig, I am an idiot. You guys probably know a lot more than me about all this stuff.

<note to everyone else who reads this thread>

If you want to help out, or want to know more about the project, please feel free to email me or anyone else on the project, or join the mailing list. At this point, we only need a few python programmers, but if you join the mailing list, youll be able to help out with docs and bug hunting.

There is no SF.net mailing list, did you guys start one and forget you didn't mention it in the thread?

Ha1f meant this one I think:
https://lists.sourceforge.net/lists/...kgswarm-develp

And the plot thickens...
:D

corbintechboy, I can't read the list archive, can you add me as a project member please?

I'm cool_walking_ on SF btw.

I think I approved you. If you have any problem let me know.

Yeah you added me to the mailing list, but you need to be a project member to read the archive (Is there anything worth reading in there?)