Free public redirect project

[[wH]Demetrios]

I got a couple of emails this morning and now have some webcam meetings scheduled tonight to discuss this project with a couple of the bigger names in UT.

I love this damn game. I always have. It got me through my divorce, through moving across the country, through being sick. That simple purchase of Unreal has given me thousands of hours of the best entertainment ever and I don't want that to end yet. A pub redirect project needs to get done, and it's going to be up to us to do it.

I'll research hosting today and find out what we are looking at. What I need from you guys is some commitments to help. Once this thing is up and running I need a rotating staff that understands FTP and redirects. In a nutshell this is how things will work...

There will be a public page where the redirect is located at, exactly like the old free redirects used to be. Everyone is able to use the same link on any servers they like and it will be fully accessible. On the same domain will be a small forum for reporting files, problems, requests, etc. I want this to be as bone simple as possible, maybe even posting without having to create accounts. I'll just need somewhere to post when someone attempts to upload duplicate mismatches, etc.

Potential new redirect files will upload to my home FTP server so they can be checked before being added to the public redirect. This will weed out anything that shouldn't be there and prevent most any 'issues' I've seen from redirects.

What I need from you is either to help judge files submitted and move them to the public repository and/or work the forum as an admin. Files would need to be moved daily (at least) from my home server to the redirect, preferably more often if several people got involved. I expect the small forum to get bombed with spammers but it needs to remain easy to use and post at so it's just going to be something that has to be tolerated and kept clean.

Mirroring UT-Files is just going to have to be someone else's thing, it's too much for me but I can damn sure try to cover the redirect problem if I can garner a little help. Is anyone interested? I would happily train you if you could commit to covering one day every couple of weeks or so.

All for UT; what you need???

[JackGriffin]

I heard back from Burst's sales rep and we chatted about things. They offered a 100MB unmetered connection on east coast US with a 250 GB capacity. I was honest in what I needed it for and he assured me it would suffice with room to spare.

Does this sound good enough to you guys?

[Dr.Flay]

Please treat this post as a boring technical guide, showing examples and reference aimed at web-developers, and people still at school.

@DrFlay: While the cloud is good, if it belongs to 1 entity you get back to the exact same problem.
I may own a cloud with lots of people and servers and whatnot, but what if I shut it all down all of a sudden? It doesn't fix that particular problem and we're back to zero.
The solution is to have mirrors in different hosts, either cloud-based or not, it doesn't matter, we just need to have redundancy in *different hosts*, either they're better or worse, it doesn't really matter, what matters first and foremost is to not actually loose the important UT-changing files.

Very good point to bring up, and a point that has been dealt with in the past after the main mirror crashed in 2004.
The Wikipedia article covers it, and is small enough to digest quickly (Shorter than my posts anyway).
http://wikipedia.org/wiki/Aminet

but that is not what I meant by "Cloud",

Spoiler

as I agree with you. I have experimented over the last few months with various "Cloud" options (even the beta Winamp now has cloud storage), but they all leave you with nothing if they die.
"Cloud" is just a marketing term to sell people something that already existed, but don't use (Old-news, new name).

DATE:1999 - I used to upload music and playlists to my first webspace, with my "pay-as-you-go" internet account, from my Amiga at home. The old MSN-Live site had various modules including Media Player. I added the playlists to my profile.
The playlists contained my files, radio stations, and links to files on other sites (inc. the entire Aminet MPG archive).
On my Amiga desktop was a shortcut to my FTP folder. I could just drag and drop files back and forth (just like you can with Windows), so I always had access to my music at work, and anywhere with internet.
As I used a standard ASX and M3U, I could play them from anywhere, with almost anything.
Over the years resources have moved from site to site, so I just edit the new locations and carry on.
Here, stream some quality tracker music from 1996 http://files.myopera.com/dr-flay/Music/ ... 996mp3.m3u

My idea of "Cloud" is sites like http://seesu.me

Main server down? My immediate solution

Spoiler

would be to temporarily use a recent Web Archive copy, or post a list of all the actual sites, until the portal is re-hosted (unless all the mirrors died). OLD EXAMPLE
This is how the original mirroring was done. You chose from a list which mirror you wanted.
The mirrors with HTTP access, all had links back to the portal and other mirrors.

Some people made their own Aminet front-ends, that would still function if they could see a working mirror.
http://www.aminet.de

How they adapted

Spoiler

After the crash, the new indexed master-site was launched with a new GUI, but still based on folder-structure and quick to navigate.
There are now only about 14 mirrors, but as long as 1 still exists, even if the portal is gone, the files are still available and can be mirrored again (and a GUI is still optional, a link will do) http://wiki.aminet.net/Mirrors
All mirrors have the same interface, no matter which URL you use eg. uk.aminet.net de5.aminet.net se.aminet.net us4.aminet.net, and are all browsable eg. http://ftp.plig.net/pub/aminet/ ftp://ftp.plig.net/pub/aminet/
If the portal dies, you move it to 1 of the mirrors.

I refer to the Aminet regularly because they went through these problems years ago, and the community solved it.
The quickest way to explain is to just go to the main portal and poke around
You don't have to download anything, but you will understand within seconds how it works. BTW there is plenty of music and pics that anyone can use as a test.
Master portal http://aminet.net/setup (I've given a link to a useful page).

What we can take from their model

Spoiler

This has worked very well since 2004, and I'm sure that with modern techniques, an even better model can be built to suit our future needs.
As it is all hosted on real FTP servers (not dumb freebie accounts), it can easily be kept FXP synchronised with the nominated master (and a master can change).
You wouldn't even have to upload all the files to a new master. You just move the portal GUI to the fastest full mirror.
How long do you think that would take you Ferali ?

Maintaining the master-server and registering the domains is the thing that costs the Aminet (there is a wider choice of domain options now).
If it was too much effort or cost, no amount of good-will would have kept the Aminet going.
It is a community project made possible, because everyone uses easy access, low maintenance, free FTP space.
Warning
You will not find adverts on Google for free storage on University servers. You have to ask your local Uni personally for "permission to store a non-commercial community project".
This may be a tall-order for most people, as you can't just sign-up like a drop-box account etc.
It will involve an actual conversation with someone emails, phone-calls etc.
If you want to find a free FTP server near you, look through the download page of a popular Linux distro.

...Unless you are currently still at school, College, or University. In this case you have no excuse not to ask the head of the IT department.

There a short post ...Shut-up, that's not cheating

Oh, and MD5 (or other) hashing should easily find the misnamed and duplicate files

[JackGriffin]

I love a good conversation and I highly value the input everyone gives. With that in mind though I'd like to suggest that paper apply his double-post policing to also helping keep threads on-topic. Several important postings have gone totally off the rails in the last couple of days and it obfuscates the rather important ideas presented. A super-important topic on server crashing devolved to a discussion on ModDB and now this thread is veered hard away from it's purpose.

Threads are free, make a new one. We'll even let you title it yourself.

Some ideas are very important and care should be taken to preserve the thought process, otherwise UT99 just becomes a jumbled mess of posts and thoughts like many of the Unreal forums I read.

Now, either talk about a free public redirect or I'm going to come through your computer screen, choke you into a nice nap, set your screensaver to Justin Bieber, eat your cookies, and buttrape your dog.

Don't make me get up. Your dog is already panting.

[UT99.org]

I heard back from Burst's sales rep and we chatted about things. They offered a 100MB unmetered connection on east coast US with a 250 GB capacity. I was honest in what I needed it for and he assured me it would suffice with room to spare.

Does this sound good enough to you guys?

You could probably share the price. I think the 100mb would be enough, and it's the same general location as skillz's one so not much will change for people but the url. Was this an open question to everybody or aimed at whoever is going to fit the bill?

[papercoffee]

@Dr.Flay
...as you can see they can't be proselytized that easy.
Keep in mind, missionaries who try to convince a tribe forcefully to something "better" will mostly be eaten...

or their dog will be buttraped.

[Wises]

if youre going ahead with this then as a community sustained service I recommend perhaps trying the following;

> setup a paypal account for donations
> build forums on same host perhaps , they come these days with domain names
> choose your OS carefully , Linux/Windows/Other
> perhaps start on a lower-plan with a host that is dynamic
> on the forums , request from all pay-pal donations the 'Iam a valid donation' API and build a small php script which stores the total (paypal accounts) balance.
~ in the forums under a special 'forum' for 'donators' and 'Honorary Members' section the balance and some info like current costs etc
~ also include upgrade prices and options for future expansion
> remove indexing from redirect server (stop leeching)
> install download limiter (php or other) which allows only 2-5 connections per peer per file. (allowing for lan players <=5)
> in the members section of the redirect server have a 'submit files script' which can be automated to even compress > move new files to the public redirect.
> have a check files / hash files class that can gather information from files submitted and a handling script which can decide weather to save, replace or alert
~ base this on downloads as well and have in a different class , a file checker which removes (or moves) files not used (downloaded or accessed) to a obsolete folder.
~ if someone attempts to upload a file which is in this folder then (simply move it back) and alert user that they do-not need to upload that file
~ Appended: if someone uploads a file which is 'Mismatched' the Handler can alert user that file is already present with different hash
* from here , it can offer to send server copy (decompressed and auto-archived via handler script) to user || several options come to mind here including offering private redirect services
* another option could be check if file being used or sitting idle , if sitting idle .. (not used) replace with new version from user.
^ This kind of latteral thinking I think... will keep the redirect fresh and holding only .. files used. (reducing HDD clutter) :end appendage

thats kinda what I would setup build tbh.

[JackGriffin]

It was aimed at those who know, really, and those who are interested in using it. I've run redirects for years but never anything at this level. My guess is that given say 50 servers and the current player base that it's enough even at peak times. I had DDOS protection added and a couple of other extras so the cost is right at $80 USD a month. I'm assuming that I'll foot the bill for this completely and that's fine. This is something that needs to be done. The sales rep assured me that upgrading if needed was not a problem so I'll cross that bridge if it comes to it.

Anything that stays under $100 monthly I can take care of. Adding private redirects is a possibility but I don't want to get into sub-leasing and money collection, etc. to help fund it. I'd like for this to remain as simple as possible.

I'll order today and hopefully get this thing going within a couple of days.

@paper: I really am a mailman so chances are it's going to be rough on the dog

@wises: Some of that I will do. I know how I want this thing to go and I'll outline it more tonight as it gels and I get it set up.

[papercoffee]

> setup a paypal account for donations

This doesn't have worked for UT-files why should it now?

[JackGriffin]

You have to go into this stuff knowing that no one is going to help financially. That's fine for my position but this is about as far as I can go for an extended time. It's plenty enough though.

[Wises]

@paper lack of spamming / marketing ability.. hence why it closed I guess.

@Jack .. one man fitting the bill is only going to delay the inevitable. another closure.. honestly $1200 year?.. thats no easy feat
unless you are going to get something back from it , is it really an ideal option?

how about sponsorship (theres bound to be a company somewhere) that could help foot the gear and take an advertising cut instead..

idk.. tbh.

[UT99.org]

@Jack Well considering how many people have lined up as interested in hosting another mirror. Scarface, myself, globalgamers. Is it you that is going to sort through the files. And what makes you think Skillz will send you them, you know he doesn't like or trust you? He hasn't been in contact with myself and I did put my hand up, I think somebody neutral will need to step in to get the files. You can't just set everything up and demand the files and then bad manner him if he doesn't want to give them to you. I'm sorry but anybody who he hasn't specifically said no to would be best to contact him over this

[$carface]

I haven't received a response from skillz yet....

With regards to DDoS protection. It's a webserver and I doubt anyone would attack it (since it's purpose is very neutral) so you could trim your bill a bit there and if you need it then purchase it.

What's the max bandwidth of protection they provide though? I wouldn't imagine it to be for the full port?

[JackGriffin]

I haven't received a response from skillz yet....

With regards to DDoS protection. It's a webserver and I doubt anyone would attack it (since it's purpose is very neutral) so you could trim your bill a bit there and if you need it then purchase it.

What's the max bandwidth of protection they provide though? I wouldn't imagine it to be for the full port?

You may not get a reply. Sadly he's getting a bit of a bad name over this and he may not want to deal with it. No matter, it can all be done without him.

I'll chat you the particulars of the server in private.

[Feralidragon]

He could have posted here if someone wanted to recover something first though (I am not sure if he asked in his forum though, but most people come here rather than elsewhere nowadays apparently, so...).
Remember L4Y? It was also one of the biggest UT resources back in the day, and once the guy reached its financial limit, he decided to shut it down, but before he did so he made sure to ask if someone wanted full zips with everything to save up.
BladeHunt from ModDB for instance (which also did some stuff for RuneStorm) was the one making the backup and posting it all in ModDB.

Anyway, Skillz is obviously free to do whatever he wants, and it was already nice of him to have it up and running, but perhaps he could have warned beforehand so people could have a chance to save most of it, but it's just my opinion for what it's worth. He could have perhaps made it self-sustainable with a proper search engine and a single ad, and I say this by own experience (and as someone said, advertising and marketing it in the forums is very important).
He could even get profit out of it with a bit more work, but again, the initiative and the time it has been running for was more than great, so complaining about how it ended wouldn't be fair for him, even if it could be handled better.

At max, one could start to gather the mods, and add the ones which are worth it to ModDB if they arent there already. Not only it's another fail-safe on the availability of the mods, as it would give tremendous exposure to UT as an active game.