LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 08-16-2007, 10:45 AM   #1
alek66
Member
 
Registered: Apr 2004
Location: Argentina
Distribution: netBSD/Linux
Posts: 256

Rep: Reputation: 30
Massive database data load


I am making a study on massive data load on databases.
I am wokring with mysql and oracle (those are my case of studies)

1st i did a program that makes random data, that generates a file full of sql inserts.

My doubts are
* how do I mesure how fast, how much did it took to insert all that data in time (min segs)

* are there some tools to do these stress testing.

*does mysql/oracle provide some built in functions for this?


I found dbmonster... but I dont quite understand how to use it.
Any information is usefull, so i thank you all in advance!
Thanks!
 
Old 08-16-2007, 05:39 PM   #2
graemef
Senior Member
 
Registered: Nov 2005
Location: Hanoi
Distribution: Fedora 13, Ubuntu 10.04
Posts: 2,379

Rep: Reputation: 148Reputation: 148
To measure how long it took to run grab the time before you start the inserts and then again when the inserts have completed. Subtracting will give you a simple idea of the time taken. However there are more things to consider. Running the program on the server, running it on a client machine (over the network), more importantly opening concurrent connections by running multiple occurrences of the program (which is a more important test than the other since it is closer to mimicking real world usage).

You would also want to look at ways of tuning your SQL, for example compare prepared statements against normal calls. After testing the inserts you can also test selects and that is a whole different story!

Finally a little digging around in the documentation of each RDBMS should show you how to grab statistics from the database itself, which can be used to measure the load and the time taken to perform certain tasks.
 
Old 08-17-2007, 08:37 AM   #3
alek66
Member
 
Registered: Apr 2004
Location: Argentina
Distribution: netBSD/Linux
Posts: 256

Original Poster
Rep: Reputation: 30
Thanks a lot. I am also looking to find some bulk load utilities.
 
Old 08-17-2007, 09:57 AM   #4
schneidz
LQ Guru
 
Registered: May 2005
Location: boston, usa
Distribution: fc-15/ fc-20-live-usb/ aix
Posts: 5,281

Rep: Reputation: 916Reputation: 916Reputation: 916Reputation: 916Reputation: 916Reputation: 916Reputation: 916Reputation: 916
Quote:
Originally Posted by graemef View Post
To measure how long it took to run grab the time before you start the inserts and then again when the inserts have completed. Subtracting will give you a simple idea of the time taken. However there are more things to consider....
cosider time

man time
 
Old 08-17-2007, 10:01 AM   #5
schneidz
LQ Guru
 
Registered: May 2005
Location: boston, usa
Distribution: fc-15/ fc-20-live-usb/ aix
Posts: 5,281

Rep: Reputation: 916Reputation: 916Reputation: 916Reputation: 916Reputation: 916Reputation: 916Reputation: 916Reputation: 916
Quote:
Originally Posted by alek66 View Post
Thanks a lot. I am also looking to find some bulk load utilities.
research for an oracle dump program in clear-text format like csv.

mysqlimport/ mysqldump can import/ export to csv.
 
Old 08-19-2007, 08:54 PM   #6
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 7.7 (?), Centos 8.1
Posts: 17,735

Rep: Reputation: 2523Reputation: 2523Reputation: 2523Reputation: 2523Reputation: 2523Reputation: 2523Reputation: 2523Reputation: 2523Reputation: 2523Reputation: 2523Reputation: 2523
Oracle's SQL Loader prog reads simple text files.
i believe there's an Oracle tool that can dump out to acceptable files also.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Processing data from a 'foreign' database with mysql, or tools to pre-process data. linker3000 Linux - Software 1 08-14-2007 08:36 PM
DISCUSSION: Network Attached Storage – An Alternative To Tape Drives In Managing Massive Amounts of Data primearray.com LQ Articles Discussion 0 04-02-2006 04:48 PM
pulling data from mysql database zuessh Linux - Software 5 05-29-2003 09:13 AM
Government Tracks Outlook Users in Massive Database m0rl0ck General 16 04-11-2003 08:18 PM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 12:31 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration