Last post Oct 24, 2016 08:26 PM by Mcnovy
Oct 24, 2016 04:53 AM|Mcnovy|LINK
I have a system that generates alot of logs. (just about 500.000 lines of logs per hour (only about 5.000 of those lines will be saved and used)
I want to be able to make some statistics with these data, to find some specific issues.
As this will be run from a Linux machine, I am thinking that MVC core would be the way to go.
Well, the question is, how should I insert the log data to the database?
1. Have either a perl script, or ASP core running on this or another machine, filter the needed data and insert it through the MVC core app, with a [httppost] in the controller, and post the data in a XML or JSON format, and let the MVC core app take care
of inserting it to the database.
(One advantage I see here, is that if I decide to change the database model, I will only need to change it in the MVC Core app which eases the maintenance)
2. Have a perl or ASP core script insert the data directly into the database
3. Have the MVC core app read the log files in a background job scheduler (if at all possible)
4. something else, that I haven't considered!
What is the most correct/best way to achieve this?
I am thinking 1, but is there something that I have completely misunderstood?
Thank you for your help
Oct 24, 2016 08:00 AM|Dmitry Sikorsky|LINK
I think you really have 2 options: just to use some shell script OR an (.NET Core) application. I would choose the first option if I am familiar with shell scripts (but I'm not). But I don't think that you really need WEB application (with web server) for
that, it is a bit overkill.
Oct 24, 2016 08:26 PM|Mcnovy|LINK
Thank you for your reply.
I can understand why you would suggest shell scripts for this,
and in fact, that is our current solution.
The thing is, that the log's aren't from the Linux OS or similar.
We have a Linux machine that is gathering all those logs, from some attached equipment.
I am making some statistics with those logs, and throwing them in an well formatted Excel worksheet (for each day), and send them to the parties (Not Linux savvy) for each of the equipment they take care of.
A webpage (A bit overkill perhaps) would give the users a nice graphical overview, with filtering etc. and the ability to see statistics over more than 1 day at a time.