Free public redirect project

[Dr.Flay]

I gave up hoping we'll get access to UT-files or getting a reply days ago.
Wave it goodbye, and plan ahead is all we can do.

Gotta get FreakinMeany to join in on this, but I keep missing him in the IRC as he's a busy chap.
He mentioned activating 2 new servers if need be.

He already has storage lined-up, so talk to him. if he can offer a better deal we should look at that.
If it can be used in parallel, even better.

I'm wondering if one of those Kick-starter project pages would be a good idea? People actively visit it looking to give money to something.
But they are geared up to offer rewards for investors, so I don't know how that would work with a project with no product.
Some people only use them as a tool to gauge public interest. Effectively though, this will be a charity.
Woh ! there's a mad idea. A charity-status project has certain advantages.
Something setup to look after future gamers. It would attract more public interest if we can say this works for everyone.
* I need to chat with the Web Archive team about that. It's their field of expertise.

Any-one else remember the conversation about using fall-back redirects ?
Can't remember what site it was on (here or Hook's I think).
Was it a mod idea? or info added to the maps ? I can't remember, but it is another option we were discussing to overcome a missing redirect.

Don't forget, CoralCDN is free, and will remove issues of heavy bandwidth. Anyone can already use it for files less than 50MB.
This is fine for U1 games, and probably most U2 games. Just add ".nyud.net" into your existing URL and it caches files for 12 hours.
Pay for storage, not for bandwidth.

I have some ideas on content delivery that may be able to feed a dynamic redirect. pulling sources from any or multiple URLs.
I've arranged a meeting with a programmer/web developer for tomorrow.
He is also a game developer, and makes medical database software, so he has the mental tool-set required.
If my idea is feasible, it would mean redirects could be serving files that have been hashed. This will also help with file version miss-matching.
He is currently involved in a game project, but if i can get him to help with the service he'll need at the end, this will be much smoother.

[Feralidragon]

Just to not loose track (and I am getting the feeling that I went a bit of offtopic back there I think, sorry about that, 00:30AM here so my brain may not be pumping much juice): how is the file structure planned for the redirect and other kinds of hosting (if any at all)?
I ask this because all the redirects I have seen had absolutely all the files in the same place, it was hideous. If you want to look for something, it was a nightmare to load the list, and another nightmare to find it.

But afaik they have to be that way so UT can find them iirc? How about doing this the right way now, with a PerlBal or something?
PerlBal is extremely fast and is used to add logic to a service in the translation and internal redirection of urls and paths.

What I am trying to say is that perhaps you can shard the files into subdirectories this time: Textures, Sounds, System, etc, and within those have some sharding (like folders starting in a/ b/ c/ d/, etc, based on the first character of the file name).
And then let PerlBal, with some regex logic there, detect any <someurl>/<somefile>.<someext>.uz and redirect it *internally* to <someurl>/<folder based on someext>/<first letter of somefile>/<somefile>.<someext>.uz, which from UT it would still be like if it was being served by <someurl>/<somefile>.<someext>.uz while having some internal organization in the server, and also some speedup (since having all the files in the same directory makes the search for it slower when trying to access it).

Example:
UT asks: http://www.redirect.com/somePackage.utx.uz
PerlBal checks the url requested:
- utx: then it goes to Textures/
- starts with "s": then it goes to s/
- internal redirect to: http://www.redirect.com/Textures/s/somePackage.utx.uz
- however UT will still see it as if he was receiving as http://www.redirect.com/somePackage.utx.uz, since from outside the server you simply don't see this redirect taking place at all

Just a small suggestion

[JackGriffin]

There's been a lot of discussion on this at UA Ferali but with mixed results reported. My own experience is that unless the files are in a singular folder en masse then the engine cannot see them. Now there are likely ways around this but it's something that needs to be heavily researched before it's depended on to work, especially across different OS versions of UT.

[Feralidragon]

But that's the thing, for UT (or any client for that matter), it will think it's receiving the file from the root folder.
This redirection is internal, so from the outside it looks like every file is in the same place, when in fact it's not internally.
Think on this like soft and hard links in a HDD in a operating system: you see the same file from N different folders, but in fact all you're seeing are different inodes pointing to the same exact file location. When you access it through UT or anything else, it sees it just any other regular file.
This isn't any different.
More, it has the advantage to actually make it possible to have more than 1 redirect server, if this gets expanded later on.

Anyway, this is something that can be done later on down the road, you can even set up the whole thing first the normal guaranteed-to-work way and then do small tests later on instead, and if it gets to be working, move all files to their respective new locations with a simple bash/python script or so, so no biggie, like I said, it's just a suggestion.
I will leave the links here in case you want to look at it later on (the text in them are very short and very straight to the point):
http://en.wikipedia.org/wiki/Perlbal
https://github.com/perlbal/

[JackGriffin]

I kinda see where this is attractive. Don't let me forget about this discussion down the road a little bit, once things are rolling along nicely.

BTW an update: I ordered and paid for the server about an hour ago. As for a time frame when this will be 'live' the rep told me today

Standard delivery time frame for a dedicated is 24-72 hours but honestly it is usually closer to the 24 than the 72 and sometimes much quicker.

I posted this at UA and BU as a sort of announcement thing:

First I'd like to thank Skillz for all that he has done for the community. UT-Files was a great resource and will be sorely missed.

One of the things that will kill UT online play is the lack of some sort of public redirect that admins with limited resources can use. UT-Files free redirect was important and it's closing really hurts.

I spoke with burst.net and purchased a package to provide free redirect service for anyone running a UT server. They are configuring the server as you read this and hopefully it will be live in the next 24 hours and available for your use.

Because this is happening so quickly I haven't had time to set up a proper gateway for the adding of compressed files. For now I have set up a private FTP server where admins can upload their redirects and then once I verify the contents they will be immediately placed onto the live redirect and be ready for use. Eventually I'll automate this but for now this will get you up and running if you need to use the service.

If you want to use this then just PM me (or drop me an email) and I'll send you FTP credentials and you can begin uploading your files. As quickly as burst has the server configured I'll begin uploading files into it (as will a couple of other admins) so within 48 hours it should be fully stocked and rolling.

Don't give up on UT if you depended on UT-Files. We can work together to overcome the loss.

[MrLoathsome]

Agree with everything Jack just said on this I think.

Once he has things up and running, and with a good core set of uz files on it, it might be good
to see if others can mirror it or something at that point.

Dr. Flay has posted some interesting plans that might indeed work very well as far as replacing
all the content that was at UT-Files aside from the public redirect.

But sadly, I have to agree with Jack on that also. It would require quite a few people to get
involved, and I just don't see that happening at this point with what is left of the UT-Community.

[Wises]

there are/were some 800+ servers for UT99 .. be interesting to see. how many were using the free redirect services of ut-files to start with
I would think that , if it affected even 20-30% then I'd expect a big rant somewhere about it?

i think the main problem is , all the files from the repository.. these may not be able to be replaced I fear.

redirects are.. not really a problem , and can be setup very easily. What we really need is to contact Leroy and see if he has a backup (I think he does)

> once a new location is setup , ask him to upload the backup.
> get someone to go through and properly organise the data , mods maps etc.

...

[Feralidragon]

Btw, you can do somewhat the same thing with Apache and the RewriteRules module (which comes already with it, but deactivated).
The thing with Apache is that is heavier and RW rules are regex only (and heavier to process too), while a PerlBal is much lighter and suitable for static content, and besides regex, you can do it through custom coding making it far more powerful for this sort of thing.

Anyway, this is just something to think on later down the road, and which could make the redirect far more resilient too if someone else adds more machines later to connect to that one.

[$carface]

If you use any type of webserver that is not litespeed, I'd suggest running apache with nginx as a reverse proxy to serve all the static files. NGNX is a great webserver and in some benches surpassess litespeed (paid solution) in serving static files (which the redirect will be).

Apache's prefork MPM do use huge amounts of RAM that easily saturate all the available RAM. But when you use the combination of Apache and NGINX the results are pretty astounding. Having this setup, Apache only processes dynamic content (like PHP) while all the static is served by NGINX. To put it into context YouTube used to use this setup.

I'd suggest though to turn on IP tracking and choking to prevent bandwidth saturation from site rippers. It's ok to site rip and create mirrors but if the bandwidth is used in one day, that's not really a satisfactory outcome, especially if you go into overages, you're going to be footing a very hefty bill. TBH though, as wises mentioned...I don't think the redirect is that much of a big deal. I mean, before the public redirect was even offered, everyone had their own arrangements for the redirect and footed a relatively small bill themselves to have their own private redirect.

Are you purchasing a managed or unmanaged solution? I can provide some help with an unmanaged server however, the time I can contribute is somewhat limited these days...but I'll do what I can.

A bit off topic here. ut-files also hosted iptocountry server which I maintained. At the same time though I provided the public one from unrealkillers.com

I've created a further two mirrors for iptocountry last night (for redundancy in case one is unavailable or for some reason does not respond) and will post these both tonight.

[Feralidragon]

I never looked into nginx, but I always heard that it's pretty good too.
Are you able to create code in it to manipulate the requests themselves there or is it limited to regex alone? If yes, in which language?

[$carface]

I never looked into nginx, but I always heard that it's pretty good too.
Are you able to create code in it to manipulate the requests themselves there or is it limited to regex alone? If yes, in which language?

You're referring to the rewrite rules right?

If so, it's not limited to regular expression AFAIK. However, I must admit, after years of being used to mod_rewrite getting used to the NGINX rewrite rules and defining my own rules to steer towards what I wanted was pretty difficult. Since I had to factor in that vBSEO was rewriting eveyrhing anyway (to ensure SEO) and then, I had the basic vB rewrites to consider to pile ontop of the NGINX rewrites.

For example something simple like this in .htaccess

Code: Select all

RewriteCond  %{HTTP_HOST}  !unrealkillers.com
RewriteCond  %{HTTP_HOST}  !www.unrealkillers.com
RewriteRule  (.*)          http://www.unrealkillers.com$1

turned into this within the actual .conf

Code: Select all

server {
    listen       80;
    server_name  unrealkillers.com http://www.unrealkillers.com;
    ...
}

server {
    listen       80 default_server;
    server_name  unrealkillers.com;
    return       301 http://unrealkillers.com$request_uri;
}

[Feralidragon]

Yes, that's somewhat what I meant to some extent.
Although processing the .htaccess rw rules is costly for Apache (specially if the right flags at the end aren't set, like [L]), since Apache is always re-processing them all, thus it's sometimes preferable to turn the .htaccess off and write the rules in actual vhost file (depending on the situation), as the vhost file is read only once and put into memory (the only disadvantage is if you change these rules constantly or must make it more flexible somehow for access control and such).

The thing is that with PerlBal (at least) you can actually write code in Perl to manipulate the requests, and Perl is one of the fastest text processors around hands down, specially with regex, hence being currently our reverse-proxy of choice (serving millions of requests everyday with no problem whatsoever).

Thus I was wondering if besides regex, you could write custom code too in Nginx and in which language, without having to add additional modules to it and whatnot.
Of course, in case of redirect files, it can be done with regex alone for what's it's worth, so either solutions are fine, but then you would need to add more rules.

[Wises]

very interesting idea's / solutions here.. it would seem we have some very talented individuals in UT-Land , and if only we could come together and make a descent living out of this game then we'd all be very happy lol

I have often thought about websites with large traffic and how to cater for this.. from what I just read above those look like the solutions. interesting.

[$carface]

Thus I was wondering if besides regex, you could write custom code too in Nginx and in which language, without having to add additional modules to it and whatnot.
Of course, in case of redirect files, it can be done with regex alone for what's it's worth, so either solutions are fine, but then you would need to add more rules.

I'm sure you could write rewrite rules within NGINX itself and I could probably (with some head banging) be able to figure some extensive rewrite rule. However, I'd need to see what the engine actually requests in the first place to come up with something to begin with. But yes, this would be possible to process with the NGINX rewrites and I don't forsee it being a problem resource wise. Having said that though...I don't know which would be more efficient and faster..

very interesting idea's / solutions here.. it would seem we have some very talented individuals in UT-Land , and if only we could come together and make a descent living out of this game then we'd all be very happy lol

Indeed, just some people are actually against receiving any sort of assistance whatsoever, for whatever reasons...

I have often thought about websites with large traffic and how to cater for this.. from what I just read above those look like the solutions. interesting.

This would definitely be your solution you could give lighttpd + nginx reverse proxy a shot, if you want a light weight webserver itself. I ran lighttpd + nginx proxy with 8 slave workers and only 1 GB of my server's RAM was utilised at peak and this also ran our multiple redirects for our servers along with the additional ones we provide to BLS and some others I won't mention

[JackGriffin]

there are/were some 800+ servers for UT99 .. be interesting to see. how many were using the free redirect services of ut-files to start with
I would think that , if it affected even 20-30% then I'd expect a big rant somewhere about it?

I think a lot of the server admins using UT-Files are not aware of the loss yet. We are because we keep up with ongoing stuff but my guess is a bunch of the lesser used gameservers are singles or doubles ran from someone's basement and they may not look in on them with regularity. It's going to take players emailing the listed owner with "How come your files are so slow?" to trigger an investigation.

They've already started to trickle in, I'm getting files uploaded as we speak.