LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 12-11-2017, 04:21 AM   #1
hhhrrrzzzzzzzzz
LQ Newbie
 
Registered: Jan 2016
Posts: 28

Rep: Reputation: Disabled
Download "real Time" Data -> Modifie it -> Upload to my Webhoster?!


Hi
I would say that my problem is very simple.
Now I work hard on my project to make an Webpage for Travellers who have no hidden Ads like the other well known.
Here the Gov offer me some "Open Data" who get updated frequently.
The major problem is I have to use the Data how the get Distributed that is in some case really weird.
Like Weather Data who get Distributed as CSV File.
The next problem like the Weather Data is there are different Source for them.
Depend on the District there are 3 Source for the Weather Data who are different in the way how the are build.

So what I am looking for is a way to run an Server on my ESXi who do there Job 24/7 unattended. The Data must be equalized and than uploadet into the DB from my Hoster to get Published on my Page.
The source can be a CSV File, XML or many other weird Formats.
Sadly the Page is just in German but here are all Open Data who I want use some of:
https://www.data.gv.at/

Before someone ask why I want just the open Data that's simple because I can do with what every I want. I relay hate 3rd Party stuff who show Ads, Track User and other ssshhttt...!
 
Old 12-11-2017, 08:06 AM   #2
TB0ne
LQ Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 22,119

Rep: Reputation: 5917Reputation: 5917Reputation: 5917Reputation: 5917Reputation: 5917Reputation: 5917Reputation: 5917Reputation: 5917Reputation: 5917Reputation: 5917Reputation: 5917
Quote:
Originally Posted by hhhrrrzzzzzzzzz View Post
Hi
I would say that my problem is very simple.
Now I work hard on my project to make an Webpage for Travellers who have no hidden Ads like the other well known. Here the Gov offer me some "Open Data" who get updated frequently. The major problem is I have to use the Data how the get Distributed that is in some case really weird. Like Weather Data who get Distributed as CSV File.
The next problem like the Weather Data is there are different Source for them. Depend on the District there are 3 Source for the Weather Data who are different in the way how the are build.

So what I am looking for is a way to run an Server on my ESXi who do there Job 24/7 unattended. The Data must be equalized and than uploadet into the DB from my Hoster to get Published on my Page. The source can be a CSV File, XML or many other weird Formats. Sadly the Page is just in German but here are all Open Data who I want use some of: Before someone ask why I want just the open Data that's simple because I can do with what every I want. I relay hate 3rd Party stuff who show Ads, Track User and other ssshhttt...!
So you're wanting a script of some type to load data into a database?? Shouldn't be a problem at all....so post what you've written/done/tried so far, and tell us where you're stuck. We're happy to help, but we will NOT write your scripts for you.

If you're looking for a starting point, you need to also provide details. Does that input data file have a way to be downloaded automatically, or are you going to do it manually? What format is that data in? What database are you using? How are you loading/doing this procedure now? Version/distro of Linux you're using? Language you want to write this in? As with your other thread, you need to provide details.
 
Old 12-11-2017, 09:47 AM   #3
hhhrrrzzzzzzzzz
LQ Newbie
 
Registered: Jan 2016
Posts: 28

Original Poster
Rep: Reputation: Disabled
Sry i am in Rush and on my phone now.
I just install Suse on my Acer Laptop that's all I have no idea which language i should choose. If some explane bow to do maybe I can start and adjust myself.
I need to know where the limits are. What doesn't a Scripting language can do?
When I am back in the Hotel I will show the data. I think it's an ordinary SQL database.
My Hoster use cpanel.
Some people from the "how to build a Webpage" Community say I should show the table on the page... Omg how dies it look? Horrible... I will make an nice page for foreign Traveller with live data such as weather who look more professional.
There are other thinks why I want grabb like public Transport Dada when I get permission for but that is a different topic.
 
Old 12-11-2017, 10:37 AM   #4
TB0ne
LQ Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 22,119

Rep: Reputation: 5917Reputation: 5917Reputation: 5917Reputation: 5917Reputation: 5917Reputation: 5917Reputation: 5917Reputation: 5917Reputation: 5917Reputation: 5917Reputation: 5917
Quote:
Originally Posted by hhhrrrzzzzzzzzz View Post
Sry i am in Rush and on my phone now.
Again, doesn't matter if you are in a rush...DO NOT use text speak. Again, it's "sorry", not "sry"
Quote:
I just install Suse on my Acer Laptop that's all I have no idea which language i should choose. If some explane bow to do maybe I can start and adjust myself.
I need to know where the limits are. What doesn't a Scripting language can do?
Since you are the one writing these programs, it is you who needs to pick the language you're comfortable with. There are MANY, including scripting languages, perl, ruby, python, C, etc. The 'limits' are whatever you're willing to write.
Quote:
When I am back in the Hotel I will show the data. I think it's an ordinary SQL database.
The INPUT data that you're downloading from that page is an entire SQL database? Great....then you (again) need to tell us what kind of SQL database it is.
Quote:
My Hoster use cpanel.
...which has no bearing on anything here. If you want to load a database/run a program, we need to know the version/distro of Linux.
Quote:
Some people from the "how to build a Webpage" Community say I should show the table on the page... Omg how dies it look? Horrible... I will make an nice page for foreign Traveller with live data such as weather who look more professional. There are other thinks why I want grabb like public Transport Dada when I get permission for but that is a different topic.
Great...so again, can you show us ANY of your efforts to date?
 
Old 12-13-2017, 07:18 AM   #5
hhhrrrzzzzzzzzz
LQ Newbie
 
Registered: Jan 2016
Posts: 28

Original Poster
Rep: Reputation: Disabled
Question

Quote:
Originally Posted by TB0ne View Post
Again, doesn't matter if you are in a rush...DO NOT use text speak. Again, it's "sorry", not "sry"

Since you are the one writing these programs, it is you who needs to pick the language you're comfortable with. There are MANY, including scripting languages, perl, ruby, python, C, etc. The 'limits' are whatever you're willing to write.

The INPUT data that you're downloading from that page is an entire SQL database? Great....then you (again) need to tell us what kind of SQL database it is.

...which has no bearing on anything here. If you want to load a database/run a program, we need to know the version/distro of Linux.

Great...so again, can you show us ANY of your efforts to date?
So I dont know any scripting languages. Which one is easy to understand and could done the Job?
To see the Data you have click on "Entdecke -> zur Resource"
Here are the source:
https://www.data.gv.at/katalog/datas...4-2d9b3defbedc -> Temp and Air Pollution for Vienna (only)
https://www.data.gv.at/katalog/datas...b-c2b3061bfe5a -> another Temp and Air Pollution Measurements
https://www.data.gv.at/katalog/datas...4-2ea8f40c746f -> Weather for selected City in Austria

for example this 3 Data should be connected together and than equalised to have an DB where all available Data are in one DB.

What did you mean with Linux Distribution? On my own Server? I dont know any so I dont pre fare any.

Are there any Question who I dont replay with the Data you need?
 
Old 12-13-2017, 08:55 AM   #6
TB0ne
LQ Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 22,119

Rep: Reputation: 5917Reputation: 5917Reputation: 5917Reputation: 5917Reputation: 5917Reputation: 5917Reputation: 5917Reputation: 5917Reputation: 5917Reputation: 5917Reputation: 5917
Quote:
Originally Posted by hhhrrrzzzzzzzzz View Post
So I dont know any scripting languages. Which one is easy to understand and could done the Job?
They are all about as easy (and hard) as each other. And ANY of them can do it.
Quote:
To see the Data you have click on "Entdecke -> zur Resource"
Here are the source:
https://www.data.gv.at/katalog/datas...4-2d9b3defbedc -> Temp and Air Pollution for Vienna (only)
https://www.data.gv.at/katalog/datas...b-c2b3061bfe5a -> another Temp and Air Pollution Measurements
https://www.data.gv.at/katalog/datas...4-2ea8f40c746f -> Weather for selected City in Austria
So you are downloading the data manually. That's fine, but you should investigate using wget or curl to try to automate this. Read the man pages on those commands.
Quote:
for example this 3 Data should be connected together and than equalised to have an DB where all available Data are in one DB.
Right...again we UNDERSTAND WHAT YOU WANT. But again, we are not seeing anything that you have done/tried to actually accomplish it.
Quote:
What did you mean with Linux Distribution? On my own Server? I dont know any so I dont pre fare any.
Simple question; if you're using Linux NOW, what kind of Linux? Red Hat 7.2? CentOS 7.4? Fedora 26? Ubuntu 16.04?? What did you install???
Quote:
Are there any Question who I dont replay with the Data you need?
AGAIN: ***YOU*** need to look at the utilities/commands that have been mentioned before, read up on their use, and START to write your own scripts. AGAIN, we are happy to help, but we aren't going to write it for you. Show some effort, and we'll help you out.
 
Old 12-14-2017, 12:23 AM   #7
ondoho
LQ Addict
 
Registered: Dec 2013
Posts: 12,901
Blog Entries: 9

Rep: Reputation: 3519Reputation: 3519Reputation: 3519Reputation: 3519Reputation: 3519Reputation: 3519Reputation: 3519Reputation: 3519Reputation: 3519Reputation: 3519Reputation: 3519
Quote:
Originally Posted by hhhrrrzzzzzzzzz View Post
So I dont know any scripting languages. Which one is easy to understand and could done the Job?
To see the Data you have click on "Entdecke -> zur Resource"
Here are the source:
https://www.data.gv.at/katalog/datas...4-2d9b3defbedc -> Temp and Air Pollution for Vienna (only)
https://www.data.gv.at/katalog/datas...b-c2b3061bfe5a -> another Temp and Air Pollution Measurements
i had a look at e.g. this: https://data.wien.gv.at/daten/geo?ve...etCapabilities
right-click on the page in your browser, select "View Source".
it's XML.
i'd use a bash script and xmllint to parse it and get at the data i need (see my weather forecast in my github).

but all this talk about databases is way beyond me.
 
Old 12-14-2017, 07:58 AM   #8
TB0ne
LQ Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 22,119

Rep: Reputation: 5917Reputation: 5917Reputation: 5917Reputation: 5917Reputation: 5917Reputation: 5917Reputation: 5917Reputation: 5917Reputation: 5917Reputation: 5917Reputation: 5917
Quote:
Originally Posted by ondoho View Post
i had a look at e.g. this: https://data.wien.gv.at/daten/geo?ve...etCapabilities
right-click on the page in your browser, select "View Source". it's XML. i'd use a bash script and xmllint to parse it and get at the data i need (see my weather forecast in my github).

but all this talk about databases is way beyond me.
That part can be fairly straight forward. If you're already parsing XML, convert it to a CSV file, and MySQL has utilities to import it already.
 
Old 12-15-2017, 12:45 PM   #9
hhhrrrzzzzzzzzz
LQ Newbie
 
Registered: Jan 2016
Posts: 28

Original Poster
Rep: Reputation: Disabled
Thanks tomorrow morning my flight Home goes and I will read it later.

How to import the Data every 30min or so?
 
Old 12-15-2017, 12:56 PM   #10
dugan
LQ Guru
 
Registered: Nov 2003
Location: Canada
Distribution: distro hopper
Posts: 9,255

Rep: Reputation: 4073Reputation: 4073Reputation: 4073Reputation: 4073Reputation: 4073Reputation: 4073Reputation: 4073Reputation: 4073Reputation: 4073Reputation: 4073Reputation: 4073
Quote:
Originally Posted by hhhrrrzzzzzzzzz View Post
Thanks tomorrow morning my flight Home goes and I will read it later.

How to import the Data every 30min or so?
Have a CRON job run the script.
 
Old 12-16-2017, 03:57 AM   #11
ondoho
LQ Addict
 
Registered: Dec 2013
Posts: 12,901
Blog Entries: 9

Rep: Reputation: 3519Reputation: 3519Reputation: 3519Reputation: 3519Reputation: 3519Reputation: 3519Reputation: 3519Reputation: 3519Reputation: 3519Reputation: 3519Reputation: 3519
Quote:
Originally Posted by dugan View Post
Have a CRON job run the script.
if your system is already running systemd, a systemd timer would be more appropriate.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Adding data upload and data download time to Squid access.log sanjeetkumar086 Linux - Server 0 08-19-2014 12:44 PM
Adding data upload and data download time to Squid access.log sanjeetkumar086 Linux - Enterprise 1 08-18-2014 10:59 AM
Using "normal", vs "low-latency" vs "real-time RT" kernels GTrax Linux - Software 7 07-10-2014 05:34 AM
Where to download precompiled bash binaries, such as "time" and "top"? elinuxqs Linux - Newbie 12 11-14-2005 09:36 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 05:42 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration