LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Programming (https://www.linuxquestions.org/questions/programming-9/)
-   -   Massive database data load (https://www.linuxquestions.org/questions/programming-9/massive-database-data-load-577564/)

alek66 08-16-2007 10:45 AM

Massive database data load
 
I am making a study on massive data load on databases.
I am wokring with mysql and oracle (those are my case of studies)

1st i did a program that makes random data, that generates a file full of sql inserts.

My doubts are
* how do I mesure how fast, how much did it took to insert all that data in time (min segs)

* are there some tools to do these stress testing.

*does mysql/oracle provide some built in functions for this?


I found dbmonster... but I dont quite understand how to use it.
Any information is usefull, so i thank you all in advance!
Thanks!

graemef 08-16-2007 05:39 PM

To measure how long it took to run grab the time before you start the inserts and then again when the inserts have completed. Subtracting will give you a simple idea of the time taken. However there are more things to consider. Running the program on the server, running it on a client machine (over the network), more importantly opening concurrent connections by running multiple occurrences of the program (which is a more important test than the other since it is closer to mimicking real world usage).

You would also want to look at ways of tuning your SQL, for example compare prepared statements against normal calls. After testing the inserts you can also test selects and that is a whole different story!

Finally a little digging around in the documentation of each RDBMS should show you how to grab statistics from the database itself, which can be used to measure the load and the time taken to perform certain tasks.

alek66 08-17-2007 08:37 AM

Thanks a lot. I am also looking to find some bulk load utilities.

schneidz 08-17-2007 09:57 AM

Quote:

Originally Posted by graemef (Post 2861362)
To measure how long it took to run grab the time before you start the inserts and then again when the inserts have completed. Subtracting will give you a simple idea of the time taken. However there are more things to consider....

cosider time

man time

schneidz 08-17-2007 10:01 AM

Quote:

Originally Posted by alek66 (Post 2861945)
Thanks a lot. I am also looking to find some bulk load utilities.

research for an oracle dump program in clear-text format like csv.

mysqlimport/ mysqldump can import/ export to csv.

chrism01 08-19-2007 08:54 PM

Oracle's SQL Loader prog reads simple text files.
i believe there's an Oracle tool that can dump out to acceptable files also.


All times are GMT -5. The time now is 12:35 PM.