LinuxQuestions.org
Review your favorite Linux distribution.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 05-09-2008, 10:06 AM   #1
Matz
Member
 
Registered: Oct 2006
Distribution: Kubuntu 9.04, Debian Etch
Posts: 178

Rep: Reputation: 31
MySQL: How to read an undelimited ASCII file?


Hi guys, I have been searching for a solution to this issue through the web, without success.

I need to read a very simple database (just one table) with MySQL but the fields of this database are not delimited by any delimiters. Of course I have a codebook telling me the columns containing each field, but I don't know how to say that to MySQL.

Does anyone have a clue?
 
Old 05-09-2008, 10:15 AM   #2
TB0ne
LQ Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 26,632

Rep: Reputation: 7965Reputation: 7965Reputation: 7965Reputation: 7965Reputation: 7965Reputation: 7965Reputation: 7965Reputation: 7965Reputation: 7965Reputation: 7965Reputation: 7965
Try something like:

mysql -u <user name> -D <database name> -e "select * from Table" > output.filename

You may have to put in a password, too, depending on your installation, if you're running this locally or over a network, etc. Read the MAN page on the mysql command for more info.

The command above will create an ouput file, containing the data. It will be something like:

--------------------------
Col1 | Col2 | Col3 | Col4
--------------------------
data | data | data | data
..
..
..


Simple matter of running that through SED or something else, to change the "|" to a comma if you need to.
 
Old 05-09-2008, 10:35 AM   #3
Matz
Member
 
Registered: Oct 2006
Distribution: Kubuntu 9.04, Debian Etch
Posts: 178

Original Poster
Rep: Reputation: 31
Sorry, I probably have not clear enough

You assume I already created the table into mysql server. But I did not, because I don't know how to do it.
I may create the table, but as far as I understand when I want to load the data into it I have to specify the field delimeter, but there's not any as fields are not delimited.

So my problem is a bit more complicated than that
 
Old 05-09-2008, 12:31 PM   #4
nathacof
Member
 
Registered: Aug 2004
Location: Bear, DE, USA
Distribution: Slackware 11, CentOS 5.2, Ubuntu
Posts: 124

Rep: Reputation: 17
Is there any way for you to re-export the source data? There really is no way to do this without specifying some sort of delimiter for your fields:

http://dev.mysql.com/doc/refman/5.0/en/mysqlimport.html
 
Old 05-09-2008, 12:46 PM   #5
forrestt
Senior Member
 
Registered: Mar 2004
Location: Cary, NC, USA
Distribution: Fedora, Kubuntu, RedHat, CentOS, SuSe
Posts: 1,288

Rep: Reputation: 99
It can also be done if you have fixed width fields or there is something IN the data that sort of shows where one field stops and another one starts.

Could you post a couple lines of example data (don't use real data), and we might be able to help.

Forrest
 
Old 05-09-2008, 12:47 PM   #6
TB0ne
LQ Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 26,632

Rep: Reputation: 7965Reputation: 7965Reputation: 7965Reputation: 7965Reputation: 7965Reputation: 7965Reputation: 7965Reputation: 7965Reputation: 7965Reputation: 7965Reputation: 7965
Quote:
Originally Posted by Matz View Post
Sorry, I probably have not clear enough

You assume I already created the table into mysql server. But I did not, because I don't know how to do it.
I may create the table, but as far as I understand when I want to load the data into it I have to specify the field delimeter, but there's not any as fields are not delimited.

So my problem is a bit more complicated than that
Well, you do have to create the table first, and that would entail knowing what kind of data you have to manage, so you can determine what data types to use, and how big to make them.

To load it, that's a different matter, and will depend on what you're using as a data source. If you've got, say, a spreadsheet, you should be able to export it as a comma-separated values file. How you import it is up to you...you can

- Write a program in bash/ksh/perl/ruby/etc., to take the CSV file and insert the records
- Use a tool like PHPmyadmin, which may (don't know), be able to do it.

The fields in the database aren't delimited..they are just separate columns. The program/whatever you use to import the data, has to be able to find them.

You'll have to get your table built first. What are you using to import your data, and what's your original data source?
 
Old 05-09-2008, 05:34 PM   #7
Matz
Member
 
Registered: Oct 2006
Distribution: Kubuntu 9.04, Debian Etch
Posts: 178

Original Poster
Rep: Reputation: 31
My data source is a simple ascii file with data. Together with it I have a codebook from where I know to which variable a column (or set of columns) refer.
The codebook is like that:
Code:
--------------------------
Columns 1-3     Variable 1
Columns 4-5     Variable 2
Columns 6-9     Variable 3
Columns 10-13   Variable 4
....
--------------------------
the first line of data is then something like this:
Code:
1112233334444......
meaning that for first observation Variable1=111, Variable2=22, Variable3=3333, and Variable4=4444
 
Old 05-09-2008, 07:08 PM   #8
AdaHacker
Member
 
Registered: Oct 2001
Location: Brockport, NY
Distribution: Kubuntu
Posts: 384

Rep: Reputation: 32
Read the documentation on the MySQL LOAD DATA INFILE command. It does support a fixed-width format, but it's not too flexible. It selects the field width based on the size of the columns in the database table. So if you have numeric columns that are shorter than the size MySQL uses, or if you want character columns longer that what's in the actual data file, you could have problems.

Personally, I'd just write a script to convert the data file into a comma-separated format (or whatever is most appropriate for your case). You could even do that with a simple call to Perl. For example:
Code:
perl -p -i -e 's/(.{3})(.{2})(.{4})(.{4})/$1,$2,$3,$4/' yourfile.txt
Once you have a character-separated file, you'll have some more flexibility with loading the file.
 
Old 05-10-2008, 05:27 AM   #9
Matz
Member
 
Registered: Oct 2006
Distribution: Kubuntu 9.04, Debian Etch
Posts: 178

Original Poster
Rep: Reputation: 31
Quote:
Originally Posted by AdaHacker View Post

Personally, I'd just write a script to convert the data file into a comma-separated format (or whatever is most appropriate for your case). You could even do that with a simple call to Perl. For example:
Code:
perl -p -i -e 's/(.{3})(.{2})(.{4})(.{4})/$1,$2,$3,$4/' yourfile.txt
Once you have a character-separated file, you'll have some more flexibility with loading the file.

That seems a really elegant solution, will try that, thanks
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
need to replace spaces with comma (ascii file) ferradura Linux - Software 3 02-02-2007 08:39 PM
ascii file read/printing in Linux ksrinivas Linux - Newbie 5 01-07-2007 08:16 AM
Revert ASCII armored file guedellas Linux - Security 1 08-09-2006 05:25 PM
line numbering in ASCII file rohr Programming 4 03-18-2005 09:14 AM
Displaying a text file in ascii?? bauld Linux - General 4 11-21-2001 07:33 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 07:56 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration