[request] XML output of scores

[Drakas]

No no not tail is a resource hog reading the whole log file in anything at a rapid pace would be.

it doesn't read the whole file as far as I'm aware...

tail itself isn't because it only spits out that last few lines but say you have tail set to show the last 10 lines and in between processing those 10 lines 20 lines get put into the log then you miss the 10 extra lines that got written while you where processing the 10 you managed to grab.
Using tail to spit the whole file out each time WOULD be a resource hog if the log file size was 10mb and larger which I see regularly just on a osok server I imagine on ctf servers and such they grow even bigger faster. Now I know processing 10 lines of a log should be pretty instant but 10 lines can be written to the log pretty fast too specially when a round changes or if some type of abuse is happening.

This is 'tail -f', not normal 'tail', if you didn't notice.

I'm not saying this would happen often but with that method there is always the possibility for data to get lost.

How? I don't think anyone would use UNIX software like tail if it were not reliable...

It's also very hard to keep track of what you have already processed since there is no unique identifier provided. If you use syslog you have a pretty unique timestamp but I imagine if some logs come in fast enough they might have the same timestamp but then again its using another external source (syslog) to collect the logs.

here you go: unique id = timestamp + the rank of the line at that timestamp

ie.

[SELECT ALL] Code:

12:13:13 x
12:13:14 y
12:13:14 z
12:13:14 x

could have unique ids:

[SELECT ALL] Code:

12:13:13:1
12:13:14:1
12:13:14:2
12:13:14:3

[Fiz]

I wasn't saying tail was unreliable I was saying the method of which it is being used is. I know what tail -f does :) I just thought you where using it as an example to see the log visually. If you are saying actually use it in your method to parse logs I am not understanding how it could be used in a more efficient way then just piping the server directly to what you are using to parse the data. Because "tail -f" doesnt stop until you make it stop anything you use to run "tail -f" with will hang forever unless you create a pipe to it thats bidirectional like I did in my php script directly with the server. I suppose it would be a good alternative if parsing the data in the php script running the server become a problem. Moving it to parsing the log with tail -f would work. It's kind of the same solution but using different data sources.

[Drakas]

I wasn't saying tail was unreliable I was saying the method of which it is being used is.

I don't see how..

If you are saying actually use it in your method to parse logs I am not understanding how it could be used in a more efficient way then just piping the server directly to what you are using to parse the data.

It's not direct, you're doing it through PHP. If anything, this may in fact hinder your server performance because it's run inside PHP (I'm not sure though)

Because "tail -f" doesnt stop until you make it stop anything you use to run "tail -f" with will hang forever

No it won't. If you pipe tail -f | xyz, then the whole thing will terminate once xyz terminates.

unless you create a pipe to it thats bidirectional like I did in my php script directly with the server.

There's no need to have a bidirectional pipe because you're not sending anything to the server.

I suppose it would be a good alternative if parsing the data in the php script running the server become a problem.

The only good alternative, in fact.

Looking at your script, your server would get killed if your PHP code failed (did you check for all the possible bugs?).
Try restarting your MySQL database when it's running.
Or maybe the SQL server runs out of the available space.
Or maybe your server gets killed by a bug (that happens often when someone finds a bug) and you can't restart your logging from the right place, or so, and so on...

[Fiz]

Yes of course when I am done contingencies for mysql failure and other things will be set up. I said from the start I am not sure what the performance loss is from doing it the way I am but I haven't noticed any yet and I've been using it at YMH for more then a week now. I've never tried piping things like you stated with "tail -f | xyz". it would be a good choice and maybe a better method for some people. I'll actually be looking at how to use piped data like that in php to see how it works. The other issue I was wondering about was log rotation that was another reason why I chose to do it the way I am. Trying to gather stats on cut up log files seemed very messy to me and ruled out an easy way to do some more advanced ladder awards like most points per game and stuff because a game may be split up between multiple logs.

PHP doing 1 task is very stable and I don't see any major pitfalls in stability in fact using php this way would make it possible to restart the server easier and remotely. You could kill the proc to kill the server then restart it again all from a web admin. I know there are many other ways you could keep track of a running server and kill/start one. Also if the server dies because of a bug and someone manages to kill it then php can restart it automatically and because the console output is processed real time there would be no need to find the right place to restart on the log parsing. As for mysql dieing or running out of space alternative "temporary" methods of storing incoming logs could be implemented until the mysql server recovers. Like I said I dont know how feasible it will become the more complicated the script gets server performance wise. But I don't see any other problems with doing it this way other then slowing the server down(if it even does). I have a few people I want to test it as I go that use VPS's to see how it effects lower end configurations which may be effected more so then a higher end dedicated server.

So back on topic can we possibly get a module api pleeeeeeeese devs? :P I think this overrall would be the best solution even though I can't write c/c++ :P It would allow for the server owner to have control over how stats are gathered and generated instead of relying on any outputted stats the devs may decide the server should supply.

[tempest]

The best solution would still be to write and run your own syslogd replacement that analyzes the log stuff as it comes in and dumps it to a DB.

Okay, okay, jk - would be slick tho :D

[Fiz]

The best solution would still be to write and run your own syslogd replacement that analyzes the log stuff as it comes in and dumps it to a DB.

Okay, okay, jk - would be slick tho :D

I was actually thinking of doing that pretending to be syslog-ng though :P Because honestly the logs come in and they have an identifier so you could essentially pretend to be syslog-ng and gather logs from many servers remotely. I still might work on something along those lines still as it might prove to be more feasible in a ladder with a larger amount of servers. The only thing that stopped me is my language of choice(php) is still a little shaky with threading and you most certainly will have to use multiple threads to handle many connections at least I wouldn't think you would. Then again I've never created a php script that really accept many connections at a time either...

[Blue_Pr!nt]

i dont like that php mehtod fix, the idea is good but thats it for me.
i had a similar setup but using perl, poe, with yaml - this allowed a web interface to control servers, start, stop, restart, config edits/reloads, all with user permissions etc...

fiz, you could actually make your own class /subfiles that when the server is compiled it would use it, this would allow more functionality and better usage. (recompiling yourself means you lose the anti-hack feature, in the near future a lib should be released so modders can recompile with the anti-hack. should you make a good enough mod and believe it to function well after testing, it can be arrange for a compile with the anti-cheat)

but the thread has strayed from the concept.. if lucas is still on the initial idea/talk we had, the idea was for the stats to be done by the server - the server log output to be changed,so it contains the stats, so that it does not need parsing, and then the ladder DB takes direct from log output - rather then the parsing in between.

lucas confirm if this is still the direction // others do you understand?

[Luc@s]

i dont like that php mehtod fix, the idea is good but thats it for me.
i had a similar setup but using perl, poe, with yaml - this allowed a web interface to control servers, start, stop, restart, config edits/reloads, all with user permissions etc...

fiz, you could actually make your own class /subfiles that when the server is compiled it would use it, this would allow more functionality and better usage. (recompiling yourself means you lose the anti-hack feature, in the near future a lib should be released so modders can recompile with the anti-hack. should you make a good enough mod and believe it to function well after testing, it can be arrange for a compile with the anti-cheat)

but the thread has strayed from the concept.. if lucas is still on the initial idea/talk we had, the idea was for the stats to be done by the server - the server log output to be changed,so it contains the stats, so that it does not need parsing, and then the ladder DB takes direct from log output - rather then the parsing in between.

lucas confirm if this is still the direction // others do you understand?

yes, server should calculates scores by itself, in another file then actual server logs. But I don't know what format we should use, and which method.

[Blue_Pr!nt]

its not a big deal, the server output from the server source is changed making this already friendly, the storage can remain as is, with a clean output you dont need parsing, how you store it is not an issue. i dont think you look at it simply, which it can be... skb already done the stats from the server - this is easy. changing the output makes the data universal without parsing > DB