Linux - GeneralThis Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
I have a little data logger running an embedded version of Linux. It logs energy data from my solar inverters and saves them in a gzipped .csv file. Previously, those files were pushed out to a third party server which would display them in a nice graphed dashboard to be used as a kiosk. That company is no longer in business so what I would like to do is process that raw data myself. I am comfortable in a Linux environment but by no means an expert.
What is the best way to do this?
I thought about pushing that data into a database and then using something like grafana to visualize it? But that seems a little over my head.
I have a little data logger running an embedded version of Linux. It logs energy data from my solar inverters and saves them in a gzipped .csv file. Previously, those files were pushed out to a third party server which would display them in a nice graphed dashboard to be used as a kiosk. That company is no longer in business so what I would like to do is process that raw data myself. I am comfortable in a Linux environment but by no means an expert.
What is the best way to do this?
I thought about pushing that data into a database and then using something like grafana to visualize it? But that seems a little over my head.
I used R (https://www.r-project.org/) for that kind of tasks. You can read csv into data frame and then operate on it. For large csv files use data.table package and for advanced graphing ggplot2. The spectrum of available diagrams is quite wide and they can be saved as jpg, png, pdf etc. You can automate the whole task and write Rscript.
Thanks for all your suggestions. So far I have the data being consolidated, formatted, and then loaded into a MySQL database. Just need to figure out how to graph it now. I've tried out your recommendation and have also played around with pandas a little, but still looking for that perfect solution.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.