LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 07-19-2010, 04:36 PM   #1
Completely Clueless
Member
 
Registered: Mar 2008
Location: Marbella, Spain
Distribution: Many and various...
Posts: 899

Rep: Reputation: 70
Question Automating data entry into MySQL database?


Hi guys.

I have set up a MySQL database which has only three fields for user-name, password and index number. But there is a LOT of data (thousands of columns) which would take too long to enter manually. I'm sure MySQL must have a file format whereby a single large file containing different data types can be successfully automatically imported and the various data elements distributed to their intended fields.

Does anyone know which characters MySQL uses as field seperators for this formatting of data, or is there a program already written which can take raw data and organize it into a SQL compatible file?
 
Old 07-19-2010, 05:07 PM   #2
fulldaykg
LQ Newbie
 
Registered: Jun 2005
Location: Florida
Distribution: Gentoo
Posts: 4

Rep: Reputation: 0
Wow... you mean a single record contains over 1000 fields?

So a quick Google gave me something like this:

Takes a tab-delimited file and imports it into a table (you must already have the schema defined though (CREATE TABLE ... ...))
Code:
LOAD DATA LOCAL INFILE '/tmp/testdata.txt' INTO TABLE testtable FIELDS TERMINATED BY '\t';
or

Same thing as above, except if you have some of those fields with quotes, it'll remove those for you as it's importing the data. (Same deal with already having a table schema)
Code:
LOAD DATA LOCAL INFILE '/tmp/testdata.txt' INTO TABLE testtable FIELDS TERMINATED BY '\t' FIELDS OPTIONALLY ENCLOSED BY '"';
Here's a link to MySQL's Reference: http://dev.mysql.com/doc/refman/5.1/en/load-data.html

(On a side note, you might want to look into ways to split that up into several different related tables. After all, that is where relational databases come in handy. Otherwise we'd all just use Excel Spreadsheets for everything, blech)
 
Old 07-20-2010, 03:52 AM   #3
Completely Clueless
Member
 
Registered: Mar 2008
Location: Marbella, Spain
Distribution: Many and various...
Posts: 899

Original Poster
Rep: Reputation: 70
Quote:
Originally Posted by fulldaykg View Post
Wow... you mean a single record contains over 1000 fields?
D'oh! I said "columns" rather than "rows" didn't I?
Sorry about that. To clarify, there is just one table with 3 fields: username, password and index number. But there are thousands of users and these three data values are each in their own separate files. I need to combine these data into a single file which MySQL can read and place each of the data elements automatically into its correct place in the table.

If I can find out which field separator characters MSQL recognizes, then I can write a short C program to create a single input file which the database will be able to interpret and understand. If I have to populate this DB by hand it'll take forever! :-(
 
Old 07-20-2010, 04:56 AM   #4
Wim Sturkenboom
Senior Member
 
Registered: Jan 2005
Location: Roodepoort, South Africa
Distribution: Ubuntu 12.04, Antix19.3
Posts: 3,794

Rep: Reputation: 282Reputation: 282Reputation: 282
You can specify which field separator etc. Check the MySql documentation for load data infile

Code:
LOAD DATA [LOW_PRIORITY | CONCURRENT] [LOCAL] INFILE 'file_name'
    [REPLACE | IGNORE]
    INTO TABLE tbl_name
    [FIELDS
        [TERMINATED BY 'string']
        [[OPTIONALLY] ENCLOSED BY 'char']
        [ESCAPED BY 'char']
    ]
    [LINES
        [STARTING BY 'string']
        [TERMINATED BY 'string']
    ]
    [IGNORE number LINES]
    [(col_name,...)]
 
Old 07-20-2010, 10:54 AM   #5
Completely Clueless
Member
 
Registered: Mar 2008
Location: Marbella, Spain
Distribution: Many and various...
Posts: 899

Original Poster
Rep: Reputation: 70
Okay, thanks for the pointers, guys. I get the general picture now.
 
Old 07-20-2010, 11:33 AM   #6
Wim Sturkenboom
Senior Member
 
Registered: Jan 2005
Location: Roodepoort, South Africa
Distribution: Ubuntu 12.04, Antix19.3
Posts: 3,794

Rep: Reputation: 282Reputation: 282Reputation: 282
If it's solved, please mark your thread as solved using the thread tools just above the first post.
 
Old 07-20-2010, 02:50 PM   #7
Completely Clueless
Member
 
Registered: Mar 2008
Location: Marbella, Spain
Distribution: Many and various...
Posts: 899

Original Poster
Rep: Reputation: 70
Wink

Quote:
Originally Posted by Wim Sturkenboom View Post
If it's solved, please mark your thread as solved using the thread tools just above the first post.
I'm afraid it won't be for a for while at least, Wim. Ideally I need to see one of these input files for myself in a text editor. Perhaps I can find a few examples on the net, but I haven't been succeeded with initial searches thus far. I'm not a professional programmer you see. Never been anything more than an abscent-minded hobbyist, so when people post these excerpts from the manuals as you did above, only about 50% of the syntax and terminology of it makes sense. Don't get me wrong though! Spoon-feeding people every little detail doesn't teach them anything. One must apply oneself!!
 
Old 07-21-2010, 12:46 AM   #8
Wim Sturkenboom
Senior Member
 
Registered: Jan 2005
Location: Roodepoort, South Africa
Distribution: Ubuntu 12.04, Antix19.3
Posts: 3,794

Rep: Reputation: 282Reputation: 282Reputation: 282
The important parts with regards to the input file are
Code:
    [FIELDS
        [TERMINATED BY 'string']
        [[OPTIONALLY] ENCLOSED BY 'char']
        [ESCAPED BY 'char']
    ]
and
Code:
    [LINES
        [STARTING BY 'string']
        [TERMINATED BY 'string']
    ]
Fields
The fields section defines what fields look like. 'terminated by' is the field separator, e.g. a comma. 'enclosed by' which character encloses the field; if your field contains the separator, the field needs to be enclosed, else the separator in the field is treated as a separator instead of as part of the field; it's often a double quote. If the fields contains one or more double quotes, this will confuse mysql and therefore they need to be escaped (hence the 'escaped by')

Code:
1,"sturkenboom, wim"
3,"pietersen, piet"
4,"double quote ""
The above is a sample csv file containing an ID (number) and a name. The fields are seperated by a comma so the ID is '1' and the name is 'sturkenboom, wim'. Because lastname and firstname are separated by a comma as well, we need an 'enclosed by'. The last record contains a double quote, so that needs to be escaped with e.g. a backslash; this needs to be fixed in the csv file so we get
Code:
1,"sturkenboom, wim"
3,"pietersen, piet"
4,"double quote \""
And the mysql statement is
Code:
mysql> load data infile 'mytesttable.csv' into table mytesttable fields terminated by ','  enclosed by '"' escaped by '\\';
Query OK, 3 rows affected (0.00 sec)
Records: 3  Deleted: 0  Skipped: 0  Warnings: 0
The result is
Code:
mysql> select * from mytesttable;
+----+------------------+
| PK | name             |
+----+------------------+
|  1 | sturkenboom, wim |
|  3 | pietersen, piet  |
|  4 | double quote "   |
+----+------------------+
3 rows in set (0.00 sec)

Lines
The lines section defines what lines look like. No experience with the 'start with' so I will skip that one. 'terminated by' is important when the input is generated on e.g. a dos/windows machine as the line terminator in dos is '\r\n' while in linux it is '\n'

So if your file comes from a dos/windows system, you (might) need to add lines terminated by '\r\n' at the end of your mysql statement
Code:
mysql> load data infile 'mytesttable.csv' into table mytesttable fields terminated by ','  enclosed by '"' escaped by '\\' lines terminated by '\r\n';
If you don't come right, please post the first few lines of your datafile as well as some of the lines that are not imported correctly. Also post the mysql statement that you have used to import.

WimS

PS assuming you're using mysql under linux

Last edited by Wim Sturkenboom; 07-21-2010 at 01:44 AM. Reason: added PS
 
Old 07-21-2010, 06:36 AM   #9
Completely Clueless
Member
 
Registered: Mar 2008
Location: Marbella, Spain
Distribution: Many and various...
Posts: 899

Original Poster
Rep: Reputation: 70
Wow! Wim, I consider myself considerably less Clueless. Thanks! I believe that ties it all up for me. If I have any problems I'll post accordingly. Thanks again.

CC.

BTW, for anyone still head-scratching, I finally found some additional useful info on the .csv file format here:
http://www.parse-o-matic.com/parse/p...ile-Format.htm

Last edited by Completely Clueless; 07-21-2010 at 06:45 AM. Reason: found additional resource
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
graphing data from mysql database fancylad Linux - Software 1 01-08-2009 12:49 AM
My SQL statement to delete an entry in MYSQL database fails in PHP ratchie Linux - Software 1 12-29-2008 02:48 AM
Processing data from a 'foreign' database with mysql, or tools to pre-process data. linker3000 Linux - Software 1 08-14-2007 08:36 PM
adding data to mysql database ? rharris72 Programming 6 01-30-2006 03:07 PM
pulling data from mysql database zuessh Linux - Software 5 05-29-2003 09:13 AM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 10:05 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration