Shell scripting to write o/p of SELECT INTO OUTFILE into files of configurable size
ProgrammingThis forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
Shell scripting to write o/p of SELECT INTO OUTFILE into files of configurable size
I have a stored procedure in MySQL that writes the output of a query to a textfile using
SELECT * FROM tbl INTO OUTFILE . I call this procedure from a Linux shell script for automated trasaction records file generation.
I want to configure this process, by defining the maximum filesize during a scheduled run.
If the filesize exceeds the limit, write the rest of data into a second file and if second file
exceeds the limit, then into a third and so on. Can anyone help??
Last edited by josepjerry; 10-13-2011 at 07:33 AM.
Aagh, PLEASE use [code][/code] tags around your code, to preserve formatting and to improve readability. The long lines force side-scrolling and make the thread almost unreadable.
Also, before I wade through all that code, care to point out just where in it you need to change things?
And did you look at the split command, like I suggested?
Thanks for your reply!!
Gone thru the split command.It works on an already generated file.
In the application requirement, the current and required funcationality are as follows:-
Current scenario:-
(i)A MySQL stored proc is called from the shell script as given in below code.
(ii)The procedure generates the transaction dump file by querying the relevant table
and using the syntax "SELECT * FROM table INTO OUTFILE <file_name>".
(iii)A single large file is generated based on volume of transactions.
(iv) A cron is scheduled to generate the files at 1 hour interval.
Required scenario:-
Required that, at any given run of the cron job, the maximum size of file
generated by the stored proc should not exceed, say 500MB. If it exceeds,
the limit, write the rest of the records into a second / third file and so on.
That is it is required to control the file generation from the time the
procedure is called.
Note:-In the code given below the place where stored proc is called is marked in bold red.
I presume the change has to be done here.
Code:
The shell script is shown below:-
#----------------------------------------------------------------------------------------
#!/bin/bash
#!/usr/bin/bsh
#TEST_HOME : Path of application home
#SDF_CONFIG.DAT : Configuration file defining variables for schema,port,host,MySQL Home,Application home,file dump path etc.
#MYSQL_HOME : MySQL Home path defined in SDF_CONFIG.DAT file
#ROOT_PSWD : Root Password as defined in SDF_CONFIG.DAT file
#MYSQL_HOST : MySQL Host IP as defined in SDF_CONFIG.DAT file
#MYSQL_PORT : Port as defined in SDF_CONFIG.DAT file
#DUMP_HOST_PATH: Path of the generated tranasaction dump file, defined in SDF_CONFIG.DAT file
TEST_HOME=/root/Test
if test -f $TEST_HOME/Program/SysConfigs/SDF_CONFIG.DAT; then
source $TEST_HOME/Program/SysConfigs/SDF_CONFIG.DAT
else
echo "Error: SDF_CONFIG.DAT not found"
exit
fi;
DG=$(date +%d-%m-%Y_%T) # date of generation
#Call MySQL Procedure that will generate the transaction files
#Required to generate file with a configured size (fetched from database or hardcoded)
#also if the file size exceeds the specified size, write into a new file (again of max size and so on till data is exhausted)
$MYSQL_HOME/bin/mysql -e 'Call transaction.offline_transaction_proc()' -u root -p$ROOT_PSWD -h $MYSQL_HOST -P$MYSQL_PORT --skip-column-names
echo "File generation Completed !!"
exit 0;
#---------------------------------------------------------------------------------------
########################################################################################
The MySQL stored procedure being called is :-
----------------------------------------------------------------------------------------
DELIMITER $$
DROP PROCEDURE IF EXISTS `tranasction`.`offline_transaction_proc`$$
CREATE PROCEDURE `offline_transaction_proc`()
BEGIN
DECLARE v_reason_code int(2);
DECLARE v_offline_file_path_name varchar(750);
DECLARE v_offline_query varchar(5000);
DECLARE v_offline_where_clause varchar(1024);
DECLARE v_csv_path varchar(512);
DECLARE v_from_date varchar(30);
DECLARE v_from_date_temp varchar(30);
DECLARE v_to_date varchar(30);
DECLARE v_current_date varchar(30);
label: BEGIN DECLARE EXIT HANDLER FOR SQLEXCEPTION SET v_reason_code = -99;
--datetime from which to generate the transaction dump
SELECT param_value INTO v_from_date
FROM configuration.configuration_detail
WHERE param_id=4;
--update param value to the datetime upto which the transactions are picked up for processing
UPDATE configuration.configuration_detail
SET param_value= now()
WHERE param_id=5;
--fetch and store in variable the datetime upto which to be fetched
SELECT param_value INTO v_to_date
FROM rt_configuration.configuration_detail
WHERE param_id=5;
--fetch the CSV path wherin to store the file
SELECT param_value INTO v_csv_path
FROM rt_configuration.configuration_detail
WHERE param_id=7;
SET v_from_date_temp=v_from_date;
--do following until the from datetime range is less than current time fetched in variable
--ie;increment the from time by one hour and see if it is less than to time, if not continue
WHILE DATE_ADD(v_from_date_temp, INTERVAL 1 hour) < v_to_date
DO
--read tranasctions from main table for writing to file
SET v_reason_code = 0;
--Prepared statement using SELECT INTO OUTFILE <filename>
--filename is dynamic based on time of generation
SET v_offline_query = "SELECT * FROM offline_transaction WHERE dump_time BETWEEN ";
SET v_offline_file_path_name = concat(v_csv_path, "/OFFLINE/OFFLINE_", DATE_FORMAT(DATE_ADD(v_from_date_temp,
INTERVAL 1 hour),'%Y%m%d.%H.%i.%s'), ".TXT");
SET v_offline_file_path_name = concat("'", v_offline_file_path_name, "'");
SET @v_offline_where_clause = concat( "'", v_from_date_temp , "' AND '",
DATE_ADD(v_from_date_temp, INTERVAL 1 hour), "' INTO OUTFILE ");
SET @v_offline_final_query = concat(v_offline_query, @v_offline_where_clause,
v_offline_file_path_name, " FIELDS TERMINATED BY ',' ");
PREPARE offline_stmt FROM @v_offline_final_query;
EXECUTE offline_stmt;
DEALLOCATE PREPARE offline_stmt;
--increment and update the configuration table to fromtime plus one hour
UPDATE configuration.configuration_detail
SET param_value=DATE_ADD(v_from_date_temp, INTERVAL 1 hour)
WHERE param_id=4;
SET v_from_date_temp = DATE_ADD(v_from_date_temp, INTERVAL 1 hour);
-- call another procedure to drop and recreate the partition corresponding to the dump time
-- from the offline_transaction table inorder to purge the data.
Call partition_proc('offline');
END WHILE; --continue loop till from time is less than to time.
--update the dumptime completion time
UPDATE configuration.configuration_detail
SET param_value=now()
WHERE param_id=6;
END;
END$$
DELIMITER ;
----------------------------------------------------------------------------------------
Last edited by josepjerry; 10-14-2011 at 01:24 AM.
Reason: comment updation
Hmm. Unfortunately, I don't have any experience with mysql. So the command reads the procedure file for the configuration details, one of which is the filename it should write the output to, correct?
What you probably need to do is rewrite that so that it outputs everything to stdout instead, which you can then send through split.
Another option might be to write a script function to set up the output file as a named pipe, then you can background the mysql call and simultaneously read the data from the pipe into split.
Either way, it looks like you're going to have to redefine the output so that it goes through split first, and generate the final output filesname(s) in the main script, rather than letting mysql do it.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.