LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 10-12-2011, 12:33 AM   #1
josepjerry
LQ Newbie
 
Registered: Oct 2011
Posts: 3

Rep: Reputation: Disabled
Shell scripting to write o/p of SELECT INTO OUTFILE into files of configurable size


I have a stored procedure in MySQL that writes the output of a query to a textfile using
SELECT * FROM tbl INTO OUTFILE . I call this procedure from a Linux shell script for automated trasaction records file generation.

I want to configure this process, by defining the maximum filesize during a scheduled run.
If the filesize exceeds the limit, write the rest of data into a second file and if second file
exceeds the limit, then into a third and so on. Can anyone help??

Last edited by josepjerry; 10-13-2011 at 07:33 AM.
 
Old 10-12-2011, 08:22 AM   #2
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Arch + Xfce
Posts: 6,852

Rep: Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037
Can we help you? Well, it depends.

How about showing us some actual code and/or specifics about what you want to do, rather than just vaguely explaining your goals?

In any case, you might want to have a look at the split command, part of the coreutils.
 
Old 10-13-2011, 07:33 AM   #3
josepjerry
LQ Newbie
 
Registered: Oct 2011
Posts: 3

Original Poster
Rep: Reputation: Disabled
code is in next reply

Last edited by josepjerry; 10-14-2011 at 12:48 AM. Reason: removed
 
Old 10-13-2011, 07:59 AM   #4
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Arch + Xfce
Posts: 6,852

Rep: Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037
Aagh, PLEASE use [code][/code] tags around your code, to preserve formatting and to improve readability. The long lines force side-scrolling and make the thread almost unreadable.

Also, before I wade through all that code, care to point out just where in it you need to change things?

And did you look at the split command, like I suggested?
 
Old 10-14-2011, 01:13 AM   #5
josepjerry
LQ Newbie
 
Registered: Oct 2011
Posts: 3

Original Poster
Rep: Reputation: Disabled
Thanks for your reply!!
Gone thru the split command.It works on an already generated file.

In the application requirement, the current and required funcationality are as follows:-

Current scenario:-
(i)A MySQL stored proc is called from the shell script as given in below code.
(ii)The procedure generates the transaction dump file by querying the relevant table
and using the syntax "SELECT * FROM table INTO OUTFILE <file_name>".
(iii)A single large file is generated based on volume of transactions.
(iv) A cron is scheduled to generate the files at 1 hour interval.

Required scenario:-
Required that, at any given run of the cron job, the maximum size of file
generated by the stored proc should not exceed, say 500MB. If it exceeds,
the limit, write the rest of the records into a second / third file and so on.

That is it is required to control the file generation from the time the
procedure is called.
Note:-In the code given below the place where stored proc is called is marked in bold red.
I presume the change has to be done here.
Code:
The shell script is shown below:-

#----------------------------------------------------------------------------------------
#!/bin/bash
#!/usr/bin/bsh

#TEST_HOME : Path of application home
#SDF_CONFIG.DAT : Configuration file defining variables for schema,port,host,MySQL Home,Application home,file dump path etc.
#MYSQL_HOME : MySQL Home path defined in SDF_CONFIG.DAT file
#ROOT_PSWD : Root Password as defined in SDF_CONFIG.DAT file
#MYSQL_HOST : MySQL Host IP as defined in SDF_CONFIG.DAT file
#MYSQL_PORT : Port as defined in SDF_CONFIG.DAT file
#DUMP_HOST_PATH: Path of the generated tranasaction dump file, defined in SDF_CONFIG.DAT file

TEST_HOME=/root/Test

if test -f $TEST_HOME/Program/SysConfigs/SDF_CONFIG.DAT; then
source $TEST_HOME/Program/SysConfigs/SDF_CONFIG.DAT
else
echo "Error: SDF_CONFIG.DAT not found"
exit
fi;

DG=$(date +%d-%m-%Y_%T) # date of generation

#Call MySQL Procedure that will generate the transaction files
#Required to generate file with a configured size (fetched from database or hardcoded)
#also if the file size exceeds the specified size, write into a new file (again of max size and so on till data is exhausted)
$MYSQL_HOME/bin/mysql -e 'Call transaction.offline_transaction_proc()' -u root -p$ROOT_PSWD -h $MYSQL_HOST -P$MYSQL_PORT --skip-column-names


echo "File generation Completed !!"
exit 0;
#---------------------------------------------------------------------------------------


########################################################################################


The MySQL stored procedure being called is :-

----------------------------------------------------------------------------------------

DELIMITER $$

DROP PROCEDURE IF EXISTS `tranasction`.`offline_transaction_proc`$$

CREATE PROCEDURE `offline_transaction_proc`()
BEGIN
DECLARE v_reason_code int(2);
DECLARE v_offline_file_path_name varchar(750);
DECLARE v_offline_query varchar(5000);
DECLARE v_offline_where_clause varchar(1024);
DECLARE v_csv_path varchar(512);
DECLARE v_from_date varchar(30);
DECLARE v_from_date_temp varchar(30);
DECLARE v_to_date varchar(30);
DECLARE v_current_date varchar(30);
label: BEGIN DECLARE EXIT HANDLER FOR SQLEXCEPTION SET v_reason_code = -99; 

--datetime from which to generate the transaction dump
SELECT param_value INTO v_from_date
FROM configuration.configuration_detail
WHERE param_id=4;

--update param value to the datetime upto which the transactions are picked up for processing
UPDATE configuration.configuration_detail
SET param_value= now()
WHERE param_id=5;

--fetch and store in variable the datetime upto which to be fetched
SELECT param_value INTO v_to_date
FROM rt_configuration.configuration_detail
WHERE param_id=5;

--fetch the CSV path wherin to store the file
SELECT param_value INTO v_csv_path
FROM rt_configuration.configuration_detail
WHERE param_id=7;

SET v_from_date_temp=v_from_date;

--do following until the from datetime range is less than current time fetched in variable
--ie;increment the from time by one hour and see if it is less than to time, if not continue 
WHILE DATE_ADD(v_from_date_temp, INTERVAL 1 hour) < v_to_date 
DO

--read tranasctions from main table for writing to file
SET v_reason_code = 0;
--Prepared statement using SELECT INTO OUTFILE <filename>
--filename is dynamic based on time of generation
SET v_offline_query = "SELECT * FROM offline_transaction WHERE dump_time BETWEEN ";
SET v_offline_file_path_name = concat(v_csv_path, "/OFFLINE/OFFLINE_", DATE_FORMAT(DATE_ADD(v_from_date_temp, 
INTERVAL 1 hour),'%Y%m%d.%H.%i.%s'), ".TXT");
SET v_offline_file_path_name = concat("'", v_offline_file_path_name, "'");
SET @v_offline_where_clause = concat( "'", v_from_date_temp , "' AND '", 

DATE_ADD(v_from_date_temp, INTERVAL 1 hour), "' INTO OUTFILE ");
SET @v_offline_final_query = concat(v_offline_query, @v_offline_where_clause, 

v_offline_file_path_name, " FIELDS TERMINATED BY ',' ");
PREPARE offline_stmt FROM @v_offline_final_query;
EXECUTE offline_stmt;
DEALLOCATE PREPARE offline_stmt;
--increment and update the configuration table to fromtime plus one hour
UPDATE configuration.configuration_detail
SET param_value=DATE_ADD(v_from_date_temp, INTERVAL 1 hour)
WHERE param_id=4;

SET v_from_date_temp = DATE_ADD(v_from_date_temp, INTERVAL 1 hour);

-- call another procedure to drop and recreate the partition corresponding to the dump time
-- from the offline_transaction table inorder to purge the data.
Call partition_proc('offline');

END WHILE; --continue loop till from time is less than to time.
--update the dumptime completion time
UPDATE configuration.configuration_detail
SET param_value=now()
WHERE param_id=6;
END;
END$$

DELIMITER ;

----------------------------------------------------------------------------------------

Last edited by josepjerry; 10-14-2011 at 01:24 AM. Reason: comment updation
 
Old 10-14-2011, 08:26 AM   #6
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Arch + Xfce
Posts: 6,852

Rep: Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037
Hmm. Unfortunately, I don't have any experience with mysql. So the command reads the procedure file for the configuration details, one of which is the filename it should write the output to, correct?

What you probably need to do is rewrite that so that it outputs everything to stdout instead, which you can then send through split.

Another option might be to write a script function to set up the output file as a named pipe, then you can background the mysql call and simultaneously read the data from the pipe into split.

Either way, it looks like you're going to have to redefine the output so that it goes through split first, and generate the final output filesname(s) in the main script, rather than letting mysql do it.
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] Randomly select a folder using Shell Scripting kapz_unlocked Linux - Newbie 7 06-21-2011 12:02 AM
Write a shell scriprt to delete files size of 56KB umwai Linux - Newbie 4 12-18-2008 12:18 AM
Desktop icon size in gnome, configurable? rgbrock1 SUSE / openSUSE 7 02-06-2008 12:38 PM
Shell scripting: How to write to multiple files? Micro420 Programming 14 05-19-2007 03:41 AM
how to burn/write files in a cd-r/rw using bash scripting jaepi Linux - Newbie 7 05-06-2007 07:50 PM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 08:23 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration