LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - Newbie (https://www.linuxquestions.org/questions/linux-newbie-8/)
-   -   Compress folders every 6 months (https://www.linuxquestions.org/questions/linux-newbie-8/compress-folders-every-6-months-4175597217/)

trickydba 01-11-2017 07:24 AM

Compress folders every 6 months
 
I generate a folder everyday consisting of different reports. I would like to compress these folders(named 0101207,01022017,etc etc) every six months. So folders named 01012017 to 06312017 would be compressed to a folder named 'Jan2017_Jun2017'. Then of course folders starting on 07012017 to 12312017 would be compressed in a folder named 'July2017_Dec2017. These folders are on a Windows shared folder. It would be ok for this automated tasks to be installed and ran from my laptop, but if it can be ran via CRON job that would be ok too, whichever is easier. Is this possible? Any help is highly appreciated

jamison20000e 01-11-2017 07:29 AM

Code:

sudo crontab -e
was my first thought...

trickydba 01-11-2017 07:43 AM

I'm not allowed to use this program, and it takes too long and I have to go through too much to get permission to run certain commands. I guess it would be better for it to run from my laptop.

jamison20000e 01-11-2017 07:48 AM

Is it Linux on your laptop because that's how to setup a CRON job (IMHO the easy way...)

trickydba 01-11-2017 08:38 AM

No, unfortunately it's Windoze(YUCK).

rtmistler 01-11-2017 08:49 AM

@trickydba, unfortunately this is not a Windows forum.

Is Linux involved here at any point? You cited cron and then you said you can't use it due to Windows.

If you wish automated tools for Windows, then please seek an alternate forum on LQ such as the Non-Nix-General forum, or give Stack Overflow a try.

As far as Linux goes, people can offer some suggestions; however you need to cite your topology of systems, tell us where Linux is being used, or is capable of being installed, or allowed to be installed, and then people can offer some solutions here.

trickydba 01-11-2017 12:15 PM

Well I just recently found out since creating this thread that I cannot create a cron job. I apologize. I guess at this point I would like to have a script that I can execute MANUALLY that can resolve my issue

273 01-11-2017 12:20 PM

Isn't this a political issue rather than a technical one?
If you're employed as a DBA and expected to back up then you need cron -- if your employer says otherwise get it in writing they don't want a backup and forget about it (whilst looking for a new job in case the imbeciles take the company down).

trickydba 01-11-2017 01:08 PM

Well the work I do is dba plus more but I just chose to have it a part of my user name just because it was the 1st thing that came to my mind

273 01-11-2017 01:10 PM

Quote:

Originally Posted by trickydba (Post 5653481)
Well the work I do is dba plus more but I just chose to have it a part of my user name just because it was the 1st thing that came to my mind

Regardless of your job title: if your employer prohibits you from doing you job then document it and move on.

trickydba 01-11-2017 01:23 PM

I understand.........still at this time I humbly ask for assistance with my issue. Is it possible in a script I can run every 6 months manually? I figure I can get the script 1st then request for the access to create a cron task. Either way, I would still need help

schneidz 01-11-2017 02:01 PM

Quote:

Originally Posted by trickydba (Post 5653316)
I generate a folder everyday consisting of different reports. I would like to compress these folders(named 0101207,01022017,etc etc) every six months. So folders named 01012017 to 06312017 would be compressed to a folder named 'Jan2017_Jun2017'. Then of course folders starting on 07012017 to 12312017 would be compressed in a folder named 'July2017_Dec2017. These folders are on a Windows shared folder. It would be ok for this automated tasks to be installed and ran from my laptop, but if it can be ran via CRON job that would be ok too, whichever is easier. Is this possible? Any help is highly appreciated

mite i suggest using better file names so that they sort better. something like year--month--date or year--day-of-year.

the script would probably call for things like tar:
Code:

tar -cvjf 2017.5.tar.bz2 /path/to/files
what have you tried so far ?

trickydba 01-11-2017 02:46 PM

I was thinking this might work, from an older thread:
Code:


 find . -name '*.DAT' -mtime -182 | xargs  tar -czvf  Jan2017Jun2017.tar.gz

I don't know if this is right but I wish it to be more dynamic in that maybe a requester pops up asking whether the save is for Jan to Jun or July to Dec. I came up with this code so far using Whiptail:
Code:

#!/bin/bash
OPTION=$(whiptail --title "Folder Compressor - By Me" --menu "Choose your date range to compress" 15 60 4 \
"1" "January to June" \
"2" "July to December"  3>&1 1>&2 2>&3)

exitstatus=$?
if [ $exitstatus = 0 ]; then
    echo "Your chosen option:" $OPTION
else
    echo "You chose Cancel."
fi

Now I would like for it to operate like this:
If I choose number 1, it will run my code(if it's right) and create the compressed filename 'JanJun2017', appending the year
If I choose number 2, it will run my code and create a compressed file 'JulyDecember2017' appending the year
showing a progress bar, then when done it shows "Compression successful!" and exits.

I know Whiptail has a 'gauge' function that displays a progress bar

rtmistler 01-11-2017 04:33 PM

What you just wrote is what I was going to suggest. To use find and tar.

I do not feel you need to use a script.

jefro 01-11-2017 04:54 PM

Notes.

I know this isn't the question asked but if I wanted to store data on a windows machine then I'd use windows folder compression tools to save data and make the information seamlessly available to all resources.

However many linux programs might be ran from limited user permissions. Tar and many other programs could be used and piped to some compression and some have switches that make renaming rather easy. Kind of the issue is how much permissions do you have in linux. Simple tar may not be allowed. You could even have some higher permissions that allow distro based programs to be run.

trickydba 01-11-2017 05:36 PM

Well I want to do exactly as I stated above. I ran it as it is and I like it! It reminds me of some stored procs I created in stinky windows! For right now I think this is one of those tasks that I prefer to do manually

trickydba 01-12-2017 07:46 AM

This is what I came up with so far. I have not ran it yet but I hoped that someone would see this ode and correct any errors:
Code:

#!/bin/bash
OPTION=$(whiptail --title "Multi-Folder Compressor - By Me" --menu "Choose the month range to compress" 15 60 4 \
"0" "CANCEL" \
"1" "January to June" \
"2" "July to December" \
"3" "Specify custom month range FUTURE OPTION" \
"4" "FUTURE"  3>&1 1>&2 2>&3)

clear

YEAR=$(whiptail --inputbox "Please input the year" 10 30 3>&1 1>&2 2>&3)

#Actions to take based on selection
{
    if [[ ${OPTION[1]} ]]; then
        #Option 1 selected
        echo "Option 1 selected"; find . -name '0101*-0631*' | xargs tar -czvf JanJun${YEAR}.tar.gz
    fi
    if [[ ${OPTION[2]} ]]; then
        #Option 2 selected
        echo "Option 2 selected"; find . -name '0701*-1231*' | xargs tar -czvf JulyDec${YEAR}.tar.gz
    fi
    if [[ ${OPTION[3]} ]]; then
        #Option 3 selected
        echo "This option is under construction"
    fi
    if [[ ${OPTION[4]} ]]; then
        #Option 4 selected
        echo "This option is for FUTURE use"
    fi
    if [[ ${OPTION[0]} ]]; then
        #Option 5 selected
        echo "You chose to cancel"
    fi

}

I would like to include a progress bar if selecting option 1 or 2.

michaelk 01-12-2017 09:06 AM

At first glance OPTION is not an array and you might want to specify the full path to your search directory and tar.gz file.

I have not played with whiptail much so can not help with its gauge box.

trickydba 01-12-2017 09:19 AM

@michaelk............How do I do this? I created this from other sources and I only have limited knowledge of java which led me to what I have now. I have another script much simpler that I figured I should start out with(the 1st code I provided is much prettier 8.) )
I created several blank folders, ranging 01012017 through 12312017. There will be other folders to cover all days in a year but I only created this for a starting point. This is the SECOND OPTION of code I'm working with that is much simpler. The folders are in the same directory:
Code:

#! /bin/bash
echo "***************** Multi-folder Compressor ****************
****************** By Me **********************

Please choose an option:

 1)Compress Jan to Jun
 2)Compress Jul to Dec
 3)FUTURE USE
 4)Disabled "

read n
clear

echo "Please provide the year "
read YEAR

case $n in
    1) echo "Compressing folders Jan to Jun "; find . -name '0101*-0631*' | xargs tar -cf JanJun${YEAR}.tar.gz;;
    2) echo "Compressing folders Jul to Dec "; find . -name '0701*-1231*' | xargs tar -cf JulDec${YEAR}.tar.gz;;
    3) echo "For future use....closing ";;
    4) echo "This option is disabled ";;
    *) invalid option;;
esac

I get this error:

Compressing folders Jan to Jun
tar: Cowardly refusing to create an empty archive

I'm lost. Any help is highly appreciated BUT I love that 1st code, with the pretty menu n colors n all! 8.))

jamison20000e 01-12-2017 09:30 AM

I can recommend [CODE][QUOTE][/QUOTE][/CODE] [B]tags![/B] :D
Code:

eg

trickydba 01-12-2017 09:31 AM

@jamison200003.........what is that? Where do I plug it in?

jamison20000e 01-12-2017 09:35 AM

Quote:

Originally Posted by trickydba (Post 5653848)
@jamison200003.........what is that? Where do I plug it in?

I forget the quick keys but if you hit "go advanced" here you get more LQ editor options to clarify your text with... ;)

jamison20000e 01-12-2017 09:39 AM

Here we go: http://www.linuxquestions.org/questi....php?do=bbcode

rtmistler 01-12-2017 09:42 AM

Excellent point jamison20000e.

@trickydba, what they are recommending is that you either use the advanced editor and properly frame code using [code][/code] tags, there are tools available in the advanced edit mode.

Further, you can also manually type these tags in, and the ones for use in LQ are located here, and discussed in the FAQ here.

Edit: Just noticed their added reply also arrived about the same time. Yes, same link.

michaelk 01-12-2017 09:44 AM

All I implied is that whiptail does not return an array so instead of ${OPTION[x]} you would use $OPTION.

The case syntax is not correct in your second script. You need to use ;; to separate each items code.

There are many ways to create menus with or without a window.

trickydba 01-12-2017 12:15 PM

I apologize all....
@michaelk............How do I do this? I created this from other sources and I only have limited knowledge of java which led me to what I have now. I have another script much simpler that I figured I should start out with(the 1st code I provided is much prettier 8.) )
I created several blank folders, ranging 01012017 through 12312017. There will be other folders to cover all days in a year but I only created this for a starting point. This is the SECOND OPTION of code I'm working with that is much simpler. The folders are in the same directory:

Code:

#! /bin/bash
echo "***************** Multi-folder Compressor ****************
****************** By Me **********************

Please choose an option:

1)Compress Jan to Jun
2)Compress Jul to Dec
3)FUTURE USE
4)Disabled "

read n
clear

echo "Please provide the year "
read YEAR

case $n in
1) echo "Compressing folders Jan to Jun "; find . -name '0101*-0631*' | xargs tar -cf JanJun${YEAR}.tar.gz;;
2) echo "Compressing folders Jul to Dec "; find . -name '0701*-1231*' | xargs tar -cf JulDec${YEAR}.tar.gz;;
3) echo "For future use....closing ";;
4) echo "This option is disabled ";;
*) invalid option;;
esac

I get this error:

Compressing folders Jan to Jun
tar: Cowardly refusing to create an empty archive

I'm lost. Any help is highly appreciated BUT I love that 1st code, with the pretty menu n colors n all! 8.))

rtmistler 01-12-2017 12:35 PM

Not sure if this is the exact problem; however June has only 30 days.

Hence 0631 is an invalid search term.

You probably won't get it, but "use your knuckles".

January, left, upper index knuckle = 31 days.
February, dip between index and second finger = 28/29 days.
March, next upper digit knuckle = 31 days.
...
July is the upper digit of your pinky, 31 days
Go back to the index finger, August is 31 days.
Finish the year.

Or just remember that all months alternate between 31 and 30 or the special case of February. And then remember that July/August are both 31.

trickydba 01-12-2017 12:56 PM

This is my final code:

Code:

#!/bin/bash

OPTION=$(whiptail --title "Multi-Folder Compressor - By Me " --menu "Choose the month range to compress" 15 60 4 \
"0" "CANCEL" \
"1" "January to June" \
"2" "July to December" \
"3" "Specify custom month range FUTURE OPTION" \
"4" "FUTURE"  3>&1 1>&2 2>&3)

clear

YEAR=$(whiptail --inputbox "Please input the year" 10 30 3>&1 1>&2 2>&3)

#Actions to take based on selection
case $OPTION in
    1) echo "Compressing folders Jan to Jun "; find . -name '0101*-0630*' | xargs tar -cf JanJun${YEAR}.tar.gz;;
    2) echo "Compressing folders Jul to Dec "; find . -name '0701*-1231*' | xargs tar -cf JulDec${YEAR}.tar.gz;;
    3) echo "For future use....closing ";;
    4) echo "This option is disabled ";;
    *) invalid option;;
esac

and I get this error:
Code:


Compressing folders Jan to Jun
tar: Cowardly refusing to create an empty archive

I chose #1

rtmistler 01-12-2017 01:02 PM

Break that down. Enter the find command manually and verify that it finds the files. Combine that with your tar command. Use "set -xv" in the command line to see what the error is, also consider using that in the script. Perhaps the find command is not working or finding the correct files and thus it outputs nothing. After all, look what the error is saying.

trickydba 01-12-2017 02:18 PM

@rtmistler......I couldn't find the issue after doing what you suggested. If you have folders created like '01012017' that go all the way to '12312017' and you wanted to tar them to one file named as such 'JanJun.tar' and 'JulDec.tar' how can this be done?
I researched doing this with files but couldn't find anything for compressing folders.

rtmistler 01-12-2017 02:34 PM

Quote:

Originally Posted by trickydba (Post 5654030)
@rtmistler......I couldn't find the issue after doing what you suggested. If you have folders created like '01012017' that go all the way to '12312017' and you wanted to tar them to one file named as such 'JanJun.tar' and 'JulDec.tar' how can this be done?
I researched doing this with files but couldn't find anything for compressing folders.

Tar will compress and also recurse down directories. My personal favorite flags for that type of operation are -cvf, where it will create, and use verbose output. Depending on the tar filename, it will compress. For instance if you say "result.tar.bz2" it will compress to a bz2 style file. If you say "result.tar.zip", it will compress to a zip style file. I haven't experimented much and usually use bz2, but also use zip for those needing to look at it easily with Windows systems even though most modern Windows will seem to recognize bz2. Other variations such as 7z I have not tried.

My best suggestion is to always stage that in a command line first to validate that you get what you desire. Don't just jump in and assume the .zip extension works, try it out and also transfer that file to a destination system where you wish to re-read it. Or whichever extension you end up choosing.

Note that there used to be a -z flag for tar, and it probably still accepts it but ignores it silently. It might be useful for you to check the version of tar you have, check the manual page for the tar you have and verify that it will work in the manners I'm describing.

trickydba 01-12-2017 02:59 PM

UPDATE...Ok I got it partially working. Here is the code again:
Code:

#! /bin/bash
echo "**************************************************************
***************** Multi-Folder Compressor ********************
****************** By Me **********************
*************** Created on January 12,2017 *******************
**************************************************************

Please choose an option:

 1)Compress Jan to Jun
 2)Compress Jul to Dec
 3)FUTURE USE
 4)Disabled "

read n
clear

echo "Please provide the year ( if you chose option 3 or 4, just hit ENTER )"
read YEAR

case $n in
    1) echo "Compressing folders Jan to Jun "; find . -mtime 182 | xargs tar -zcvf /export/home/TESTING/JanJun${YEAR}.tar.gz;;
    2) echo "Compressing folders Jul to Dec "; find . +mtime 182 | xargs tar -zcvf /export/home/TESTING/JulDec${YEAR}.tar.gz;;
    3) echo "For future use....closing ";;
    4) echo "This option is disabled ";;
    *) invalid option;;
esac

When I select #1, I get this error:
Code:

Compressing folders Jan to Jun
tar: Cowardly refusing to create an empty archive

But when I select #2 it works, but looking at the output it seems to have compressed these folders twice:
01062017
01042017
01032017
01012017
01022017
01052017
which of course nothing should have been compressed.

Any help is highly appreciated!

michaelk 01-12-2017 03:55 PM

-mtime 182 finds files modified exactly 182 days ago. Since there is not an option +mtime your results are not valid.

-mtime n (n*24 hours)

+n for greater than n,
-n for less than n,
n for exactly n.

If you want to go by mtime you will need to specify a range but you need to calculate the days based upon the current date. If today was 12/31 then something like the following would find files between Jan 1 - Jun 30 (I did not calculate exact days...)

-mtime +182 –mtime -365

You can find files based upon the file name using wildcard expansion. Although it isn't pretty and others might have a better idea.

find . -maxdepth 1 -name "0[0-6]*"

To specify a year
find . -maxdepth 1 -name "0[0-6]??YYYY"

Would find all files/directories in the current working directory from Jan - Jun assuming they were named MMDDYYYY.

For July-August
find . -maxdepth 1 -name "0[7-9]*"
For Sept-Dec
find . -maxdepth 1 -name "1[0-2]*"

July-Dec would need 3 commands i.e two to create append the tar file and a separate command to compress.

I know that some of these commands can be confusing and in most cases you can find decent help and examples.

trickydba 01-13-2017 06:52 AM

Too confusing to me. I came up with a better idea. How can I compress folders( named like 01012017,02012017,etc ) based only on the first 2 digits (the month)?

Like for instance to only compress folders where the range is from 01****** to 06******

trickydba 01-13-2017 07:24 AM

Would this work?
Code:

echo "Compressing folders Jan to Jun "; find . "01*" ! "06*" | xargs tar -zcvf /dir1/dir2/dir3/TESTING/JanJun${YEAR}.tar.gz;;

rtmistler 01-13-2017 07:32 AM

Quote:

Originally Posted by trickydba (Post 5654316)
Would this work?
Code:

echo "Compressing folders Jan to Jun "; find . "01******" ! "06******" | xargs tar -zcvf /dir1/dir2/dir3/TESTING/JanJun${YEAR}.tar.gz;;

I believe one * should be sufficient. If you wish to mandate that you have exactly 6 digits following the 01 or 06 terms, there are regex expressions which you can use; however you'd also have to account for additional non-digits following the remainder of the dates.

trickydba 01-13-2017 07:42 AM

This is the code I used:
Code:

echo "Compressing folders Jan to Jun "; find . "01*" ! "06*" | xargs tar -zcvf /dir/dir/dir/TESTING/JanJun${YEAR}.tar.gz;;
ans this is the error I got:

Code:

Compressing folders Jan to Jun
find: paths must precede expression: 06*
Usage: find [-H] [-L] [-P] [-Olevel] [-D help|tree|search|stat|rates|opt|exec] [path...] [expression]
tar: Cowardly refusing to create an empty archive


rtmistler 01-13-2017 07:54 AM

There's a syntax error in your find command. I know there are shortcuts; however I always use:
Code:

find . -name "01*"
and you cannot use the AND to combine search terms.

This would either be six find commands, one for each month, or using a -since or date qualifier to capture the files since your last backup.

Once again, I know there are highly particular requirements. I'm remembering that this is to back up information you've generated on the behalf of a company you work for in some manner.

Personally I'd reorganize files once they'd been backed up. Make my backup and then move the backed up files to an "old" sub-directory or something similar.

If things are so restricted that you cannot move files, such as you're not allowed to move files, and this company is so very highly specific about everything, then I question whether or not you should be keeping any copies.

I've worked as a contractor for highly restrictive employers, such as government contractors, and you're not supposed to take any IP out of their shop at all. Not allowed. Then again, the security restrictions were sufficient such that if I tried to take out a drive of any type, it would be a problem, let alone bring in anything capable of storage.

If this is a commercial company, or something more serious such as aviation or power grid, things like that where once again they have protected systems. Same thing. I would not be keeping any personal records.

This "sounds" as if you're generating status reports and you wish to keep a copy. So ... keep a copy on another system, this laptop you mention, and back that up however best works for you.

michaelk 01-13-2017 08:35 AM

As stated you have to specify each month which can be done using or. Not tested but should work.

Code:

find . -type d \( -name "01*" -o -name "02*" -o -name "03*" -o -name "04*" -o -name "05*" -o -name "06*"\)

trickydba 01-13-2017 08:59 AM

I tried this code and nothing returns but I do not want to use OR, I want to pick the folders from Jan(01*) to Jun(06*):

Code:

echo "Compressing folders Jan to Jun "; find . -type d \( -name "01*" -o -name "02*" -o -name "03*" -o -name "04*" -o -name "05*" -o -name "06*"\) | xargs tar -zcvf /dir/dir/dir/TESTING/JanJun${YEAR}.tar.gz;;
I also tried this code;
Code:

echo "Compressing folders Jan to Jun "; find . -type d "01*" ! "06*" | xargs tar -zcvf /dir/dir/dir/TESTING/JanJun${YEAR}.tar.gz;;
and got this error:
Quote:

Compressing folders Jan to Jun
find: paths must precede expression: 01*
Usage: find [-H] [-L] [-P] [-Olevel] [-D help|tree|search|stat|rates|opt|exec] [path...] [expression]
tar: Cowardly refusing to create an empty archive

michaelk 01-13-2017 09:30 AM

In my previously post there was not a space between the last " and \ i.e. "06*"\) vs "06*" \) which caused a syntax error.

Code:

$ ls
01012017  02022017  03012017  04012017  05012017  06012017  07012017  11012017

Code:

find . -type d \( -name "01*" -o -name "02*" -o -name "03*" -o -name "04*" -o -name "05*" -o -name "06*" \)
./05012017
./03012017
./02022017
./06012017
./01012017
./04012017

If the current working directory contains the subdirectories as named above the command returns the results as listed.

Code:

find . -type d \( -name "01*" -o -name "02*" -o -name "03*" -o -name "04*" -o -name "05*" -o -name "06*" \) | xargs tar -zcvf testme.tgz
Completed successfully and listing its contents.
Code:

tar -tf testme.tgz
./05012017/
./03012017/
./02022017/
./06012017/
./01012017/
./01012017/f1.txt  (empty file)
./04012017/


trickydba 01-13-2017 09:56 AM

After placing .xlsx files in the months of '02', '01' and '06' it appears that the code below works:
Code:

find . -type d \( -name "01*" -o -name "02*" -o -name "03*" -o -name "04*" -o -name "05*" -o -name "06*" \)
Thank you everyone for your help

TinkGirlcc 01-22-2017 11:34 PM

I just think everything is possible


All times are GMT -5. The time now is 05:03 AM.