LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices


Reply
  Search this Thread
Old 10-09-2001, 08:12 AM   #1
ascii2k
Member
 
Registered: Dec 2000
Location: Indiana
Distribution: Red Hat 7.3/Mandrake 8.2/9.0
Posts: 154

Rep: Reputation: 30
Help! Need Suggestions!!!


OK. I have stumbled across what could be a HUGE break for Linux at my employer. The need has arisen for a rather large server running a SQL database. My employer automatically of course assumed they would use Microsoft's product. I mean it was only gonna cost them $10,000+ . Sooooo, I spoke up and mentined that Linux had something called MySQL that was free and supposed to be as good as Microsoft's product. They aren't convinced yet, but I have permission to setup up test machine(s) until the real server gets here. Now the questions...This is going to be a rather large database. They want 120 days worth of data to be available to be queried. That is A LOT of data.

!.) IS MYSQL CAPABLE OF DOING SOMETHING LIKE THIS???

Of course I want to believe it is, but I have never seen MySQL under a load so I don't know.


2.) Which distribution would be best for this?

I am currently using Mandrake 8.1 but I am leaning toward Redhat for the SQL servers. Any suggestions would be appreciated.


3. ) This is more of a MySQL question but oh well. Once the machines are ready I have to get all of this data and somehow IMPORT it into MySQL. This is RAW data we are talking about. I am praying that there is a tool for MySQL to do this but I don't know of one. I can probably get the field info for the data but I need a way to get it into a database.



I was also thinking of doning something like a cluster for this. Currently the best machine I have is a 700mhz Athlon. I also have a PIII-500. Would it be worth clustering them to balance the load?

Thanks again and ANY input is appreciated!!
 
Old 10-09-2001, 10:21 AM   #2
mcleodnine
Senior Member
 
Registered: May 2001
Location: Left Coast - Canada
Distribution: s l a c k w a r e
Posts: 2,731

Rep: Reputation: 45
Just my $.02...

I think the case could be made over availability and speed over price. You could go with a Linux/Solaris + Oracle solution or some _really_ good hardware + Linux + MySQL/PostgreSQL.

If your're looking at a quick and dirty setup you might want to just install the MySQL or Postgres RPM included with your distro just so you can tinker with it and do some show-and-tell.

I have a hunch this could break out into a full-blown flame war over which open source db server is the best, not to mention which distro.

As for importing your data it would depend a lot on what format it's in right now. You might be better off exporting from the current app into a format you can easily use/import to the chosen platform.

Last edited by mcleodnine; 10-09-2001 at 10:24 AM.
 
Old 10-09-2001, 10:36 AM   #3
ascii2k
Member
 
Registered: Dec 2000
Location: Indiana
Distribution: Red Hat 7.3/Mandrake 8.2/9.0
Posts: 154

Original Poster
Rep: Reputation: 30
Well thanks for the reply. Here are my thoughts.

Yes, a Solaris/Oracle soultion would be nice but I think that is less likely to happen than Linux. We don't have anyone that knows Oracle here and Sun servers aren't exactly inexpensive.

IF, by some miracle, we decide to use Linux, I am SURE it would be on a NICE server..probably the same server that we would run Win2k/MS SQL Server on. So that's not a problem. If we did decide to use Linux maybe I could convince them to get a copy of Red Hat server. That would be nice. I guess your are right. It doensn't matter what dist I use to test.. I am really just wanting to see if we could even do it with Linux.



I think my main problem is going to be importing the data. As I said theis is raw data coming straight from the collection device. It is in a text file but that's about it. It seems that MySQL does have an import function but how well that will work is yet to be seen. Unfortunately I am pretty sure the app that collects the data will be of absolutely no use with exporting to a more usable format. The software is old and runs on ancient SCo unix boxes to give you an idea.

So I guess I am going to try to get 2 Mandrake 8.1 machines in a MOSIX cluster and then start trying to get some data. I will let you know how things go.
 
Old 10-09-2001, 10:39 AM   #4
ascii2k
Member
 
Registered: Dec 2000
Location: Indiana
Distribution: Red Hat 7.3/Mandrake 8.2/9.0
Posts: 154

Original Poster
Rep: Reputation: 30
Oh yeah.. The last thing I want to do is start a war over which dist is best. I am just looking for comments from anyone who may have used MySQl for a large project.

Thanks.
 
Old 10-09-2001, 11:12 AM   #5
entm
Member
 
Registered: Oct 2001
Location: Atlanta, GA
Distribution: Red Hat 7.0
Posts: 45

Rep: Reputation: 15
I just picked up a book on postgres this past weekend. Right now, it looks like postgres supports transactions and sophisticated table lookups that MySQL hasn't implemented yet. My general feeling is that postgres comes much closer to Oracle than MySQL, and is a little more difficult to learn.
 
Old 10-09-2001, 05:30 PM   #6
isajera
Senior Member
 
Registered: Jun 2001
Posts: 1,635

Rep: Reputation: 45
i'm only a bit familiar with db programming, but i'd have to vote for postgres over mysql. i've used both of them for various assignments, and mysql impressed me at first, but it's not as fully developed as postgres. on the otherhand, mysql is *very* fast...
so if you're not doing very complex db searches and what not, mysql might be the way to go. postgres, however, is closer to what most ms-windows db programmers are used to.

as for the importing, you'll probably need to write a small prog to do that for you... but if it's anything like what i think it would be like, it wouldn't be that hard to write. especially if it's just in a text file... heck... that almost sounds like one of my old assignments .

Last edited by isajera; 10-09-2001 at 05:39 PM.
 
Old 10-11-2001, 08:44 AM   #7
ascii2k
Member
 
Registered: Dec 2000
Location: Indiana
Distribution: Red Hat 7.3/Mandrake 8.2/9.0
Posts: 154

Original Poster
Rep: Reputation: 30
Unfortunately, I can't take time now to learn PostgreSQL from scratch. I have to stick with MySQL.

Anyway,
I have a 3 machine linux cluster now setup and running. I am now stuck on what I was afraid would give me trouble. That is importing the actual data into mysql. The data fiel that I have is very raw. Meaning the only kind of formatting in the file is a Linefeed at the end of each line. Each record is 293 chars and there are over 100 fields in each record. So as you can see this is not fun. What I need is a (reliable) tool that will let me enter the field definitions and them import the data. The best thing I have found so far is a app called DB Tools. It seems to do what I need but it will not let me import the data. It gives me an error.


Does anyone have any ideas?


Thanks.
 
Old 10-11-2001, 05:54 PM   #8
isajera
Senior Member
 
Registered: Jun 2001
Posts: 1,635

Rep: Reputation: 45
100 fields in 293 chars? sounds like a lot of booleans...

one question: exactly how raw is raw here? is it raw in that there are no field delimiters, or that there's nothing but strange looking characters and beeping sounds when you cat a file?

if the data is in a form that's readable without knowing some strange formatting, then it's not that hard to write a program that will read a file and then put the info in a mysql database. i'm not too busy right now, and if you email me some sample data and the record information, i could probably write you a program that will at least get you started.

however, if this is data that's stored in some weird esoteric unreadable form, then you'll need to find a conversion program first, and i personally don't know of any.
 
Old 10-12-2001, 12:12 PM   #9
ascii2k
Member
 
Registered: Dec 2000
Location: Indiana
Distribution: Red Hat 7.3/Mandrake 8.2/9.0
Posts: 154

Original Poster
Rep: Reputation: 30
You were absolutely right. There were a ton of 1 character fields. Having that many fields at all is crazy if you ask me..but no one did so oh well.. Anyway, when I said 'RAW' I just meant that the fields had no delimiters. As far as I can tell there aren't any strange characters in the file..only numbers.


I really appreciate your offer and if I hadn't already wrote something myself I would accept your offer. I just wrote a small program in Pascal(Gasp!) that reads the data file and inserts a '/' between each field. I was then able to get mysql to accept the data with no problems. So that's one problem solved. The next problem is that there is a field 7 chars wide that represents a duration(4char-min 2char-sec 1char-tenth/sec). Of course this is not how they want the data to be formatted when they retrieve it. So I am going to have to use something to query the database and then format the data. We have a few ASP programmers and as much as I hate to say it, we may have have to use that.


Thanks again to everyone for their comments/help.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Any suggestions ? Slimster Debian 3 10-02-2005 01:30 PM
Any suggestions? coasterfreak212 Linux - General 5 06-29-2005 03:58 PM
LQ - Any Suggestions jeremy LQ Suggestions & Feedback 74 11-01-2004 08:54 AM
Suggestions please :) kith Linux - General 8 05-13-2004 09:45 PM
Suggestions please? paulrayner Linux - Software 5 04-21-2004 07:11 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - General

All times are GMT -5. The time now is 01:19 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration