Just got knocked back to noob status, I did.

I wanted to create my own log files for a specific purpose. Knowing "just enough to be dangerous" and observing that logs lived in /var/log, I created scripts to make and maintain entries in my log which I called /var/log/$HOSTNAME"_"$(date +"%Y-%m-%d").log, thinking myself rather clever.

Apparently that has thrown a monkey wrench into some system process, because now that I've seen the errors of my ways and called them something other than .log, the system is still working hard to remind me how foolish I really was.

Daily at midnight it creates a 0 length echo of every .log file instance I'd ever created.

These now number several dozen. My own user logging is working well now, but these useless echos get recreated daily despite my efforts to purge them. Obviously some cron job has "noticed" my errant logs and decided it has to do something to maintain them. Not so. Cease and desist already. "Would all Great Neptune's oceans wash this blood clean from my hands?"

So the question is, "How do I clean up my mess?"

(This is Armbian Buster on an Orange Pi One.)

Learn how cron.hourly, cron.daily, cron.weekly, & cron.monthly work with logrotate then configure them to work the way you wish with your logs. Those work together to manage log rotation which creates your "echo"s.
Depending on how large and how fast your logs grow you can set them up to rotate daily, weekly, less frequent, or not at all.

The whole idea of log rotation is to keep the logs of a manageable size and dated for reference into the older ones.

Uhm, yeah. I had assumed the processes at work were trying to re-init each date-named instance of my log files for future use, and that if I knew more about Linux I could "configure" whatever it is to stop.

The problem is that I never touched it in the first place. Something has reacted to the presence of my individual .log file instances, and automatically rolled that into its own processes. I don't need to modify or use those processes. I just need them to stop.

I am assuming that each hostname_yyy-mm-dd.log file name instance has been rolled into a table of "log files" somewhere. Purging the .log files does not make them go away permanently; they all come back at midnight. I have not succeeded in finding the table where those name instances are stored.

The help available seems to specify how to set up logging, not how the actual maintenance processes work. I was hoping for a little more guidance on that front.

No, nothing has interfered with your creation of the logs. Nothing has interfered, nor is anything doing something abnormal. Do just a bit of research as I suggested about how the system rotates logs and you will find out exactly what is happening and how to change it. No "table of log files" exists. Action is taken on existing files.

Where you put the log is correct. It is being rotated as configured. It is up to you to make the configuration changes to suit your needs. That means you need to learn how, or live with the current situation. An alternative would be to give me (or any other administrator) access to your machine, pay for my time to do the job for you, and have me make the desired changes.

Just one quick suggestion if I am understanding you correctly. I think you said there are files you no longer use that are empty and these are getting rotated as well. If so then delete every instance of that particular file and then new ones will not be created. For logs you are still using but the rotation does not satisfy you then my suggestion to learn how and make changes to the rotation of those files still applies.

Another hint:
Try a quick search for "how to use logrotate on linux"

Yeah I'm not making myself clear. Sorry, my bad.

I'm done with the files. Their purpose has ended. I want them out of my life.

But every time I delete them, (wholesale,... every instance), they DO come back from oblivion, all of them, at midnight.

For instance, I type: rm Valiant*.log, and they all dutifully disappear.

Yet the next day they are back.

-rw-r--r-- 1 root root 0 Mar 30 00:00 Valiant_2020-11-16.log
-rw-r--r-- 1 root root 0 Mar 30 00:00 Valiant_2020-11-17.log
-rw-r--r-- 1 root root 0 Mar 30 00:00 Valiant_2020-11-18.log
-rw-r--r-- 1 root root 0 Mar 30 00:00 Valiant_2020-11-19.log
-rw-r--r-- 1 root root 0 Mar 30 00:00 Valiant_2020-11-20.log
-rw-r--r-- 1 root root 0 Mar 30 00:00 Valiant_2020-11-21.log
-rw-r--r-- 1 root root 0 Mar 30 00:00 Valiant_2020-11-22.log
-rw-r--r-- 1 root root 0 Mar 30 00:00 Valiant_2020-11-24.log
-rw-r--r-- 1 root root 0 Mar 30 00:00 Valiant_2020-11-25.log
-rw-r--r-- 1 root root 0 Mar 30 00:00 Valiant_2020-11-26.log
-rw-r--r-- 1 root root 0 Mar 30 00:00 Valiant_2020-11-27.log
-rw-r--r-- 1 root root 0 Mar 30 00:00 Valiant_2020-11-28.log
-rw-r--r-- 1 root root 0 Mar 30 00:00 Valiant_2020-11-29.log
(...ad nauseam)

In order for cron.daily/logrotate to do that, I project it must have squirreled away every instance of a log file name I had ever created. Otherwise it could not know what filename to initialize for my ungrateful benefit.

A coward's way out would be to cron a re-purge every 00:01, but that's icky.

That really is strange.
Try looking at the config files /etc/logrotate.conf and in /etc/logrotate.d/ for some possibilities.

I have never heard of logrotate restoring deleted files like that, so I am now at a total loss. Logrotate takes existing files, as configured, and rotates them out to backups creating a new empty file to start over.

Is it possible that you configured the log rotation in logrotate and made an error in how it was set up? Is it possible that your program that created the logs at the beginning has something to do with saving old copies and putting them back? (just grasping at straws here.)

You are using armbian buster on a pi one, I have used raspios (which is essentially the same, based on debian buster) current on a pi 4B, and several other distros on the pi as well as on regular PCs for many years and have never seen nor previously heard of what you describe on any system. Maybe someone else has a clue and can chime in.

That's the really scary part--I never did set anything up with logrotate. There are no entries in /etc/logrotate.d pertaining to this, and my home cooked routines are pretty simplistic and not responsible either.

Just had a thought though,...

Is there a "default" rotation scheme that operates on every unspecified file it finds in /var/log/*.log and in the absence of an explicit coding? And does it "load" the elusive table at boot time?

One thing I haven't explicitly tried is to reboot after purging my files. If the mysterious list I've been looking for is in memory, doing that should kill it.

I'll see tomorrow if that has worked,...

I'm losing my mind.

• Purge the files with rm Valiant*.log
• Check with ls -la, and yes they are gone.
• Reboot.
• The files are back again, still timestamped 00:00 this morning.

/etc/logrotate.conf contains the default config and that is weekly.

As I said earlier, I have never seen anything restore previously deleted log files.

Just another twig to pluck. Is there a chance that you have some backup/restore process running that could do that?

Contents of /etc/logrotate.conf are virginal:

# see "man logrotate" for details
# rotate log files weekly
weekly

# keep 4 weeks worth of backlogs
rotate 4

# create new (empty) log files after rotating old ones
create

# use date as a suffix of the rotated file
#dateext

# uncomment this if you want your log files compressed
compress

# packages drop log rotation information into this directory
include /etc/logrotate.d

# system-specific logs may be also be configured here.

Similarly, there is no relevant entry in /etc/logrotate.d:

total 76
drwxr-xr-x 2 root root 4096 Mar 31 14:57 .
drwxr-xr-x 118 root root 12288 Mar 31 14:58 ..
-rw-r--r-- 1 root root 120 Mar 31 14:57 alternatives
-rw-r--r-- 1 root root 173 Mar 31 14:57 apt
-rw-r--r-- 1 root root 79 Mar 31 14:57 aptitude
-rw-r--r-- 1 root root 99 Mar 31 14:57 armbian-hardware-monitor
-rw-r--r-- 1 root root 139 Mar 31 14:57 btmp
-rw-r--r-- 1 root root 169 Mar 31 14:57 chrony
-rw-r--r-- 1 root root 181 Mar 31 14:57 cups-daemon
-rw-r--r-- 1 root root 112 Mar 31 14:57 dpkg
-rw-r--r-- 1 root root 501 Mar 31 14:57 rsyslog
-rw-r--r-- 1 root root 891 Mar 31 14:57 samba
-rw-r--r-- 1 root root 677 Mar 31 14:57 speech-dispatcher
-rw-r--r-- 1 root root 178 Mar 31 14:57 ufw
-rw-r--r-- 1 root root 235 Mar 31 14:57 unattended-upgrades
-rw-r--r-- 1 root root 305 Mar 31 14:57 winbind
-rw-r--r-- 1 root root 154 Mar 31 14:57 wtmp

No, no restore processes running either. And no other users. This is a "toy" Pi computer I have running out in the shed, solar powered, doing processing for BOINC just for jollies.

I was wrong about the datestamp--today is the 31st. They are stamped yesterday at midnight. If that matters,...

I noticed all those files are owned by root, and I assume you did the rm as root or you would have gotten an error.

The really strange thing is you removed them, then rebooted and they returned.



Here is one thing else that you might check.
It is a known fact that flash memory (usb thumb drives, SD cards, etc.) usually fail in read only mode and changes can no longer be made to the device although it is still readable.
I guess it is possible that the SD card may have failed so that changes you make appear to have worked in memory because of cached writes, but do not actually make it to the physical device. Later after the buffers clear you still see the files.
One of the things that leads me to suspect this is the fact that all the files in /etc/logrotate.d have todays date, and that is definitely not correct unless you just copied them from somewhere else. Also, the log files that keep reappearing all have the same date/time displayed in ls even though the date stamp in the file name is different.

Could you test that by copying the content of the active SD card to a new card and booting from that. If the files are still there then delete them and see if another reboot brings them back.

Yes, the rm was performed by root.

I was assuming that the boot process somehow touches all the files in /etc/logrotate.d, and thus hauls the date/time stamp up gets hauled up to the time of the boot. But hmm,... that doesn't seem to be true on my Debian 8 desktop image.

However it IS true on my other SBC, also an Orange Pi running Armbian. This one has the same log issues, so I don't think it's directly related to the general flakiness of SD cards and their kin. I've got two cases of the same behavior.

But wait, what's this? Now Valiant's timestamps have changed.

etc.
-rw-r--r-- 1 root root 0 Mar 31 19:30 /var/log/Valiant_2021-03-18.log
-rw-r--r-- 1 root root 0 Mar 31 19:30 /var/log/Valiant_2021-03-19.log
-rw-r--r-- 1 root root 0 Mar 31 19:30 /var/log/Valiant_2021-03-20.log
-rw-r--r-- 1 root root 0 Mar 31 19:30 /var/log/Valiant_2021-03-21.log
-rw-r--r-- 1 root root 0 Mar 31 19:30 /var/log/Valiant_2021-03-22.log
-rw-r--r-- 1 root root 0 Mar 31 19:30 /var/log/Valiant_2021-03-23.log
etc.

Curiouser and curiouser,...

The machine was doing nothing special at 7:30. (I was having dinner.)

Neither is there anything in the root crontab that would account for it.

Something is seriously spooky with these little guys. It's not a huge issue, but it is a perlexing one, to me.

Something is not right there, but I have no clue what.

None of my systems have ever synchronized all the dates & times for files like that regardless of the hardware they are on. My only guess was what I said earlier about a card failing, but seems unlikely that 2 different devices would do the same thing unless it was a software issue of some sort, maybe related to an update that went south.
You should see a variance in date & times on files in almost every directory.

I am still leaning toward potential card failure and suggesting at least a test with using a new SD card, just in case.

A test with another card will have to wait a few days for me to have physical access to the machine. It is in the back of the shed which is still full of all our summer deck furniture.

On both (very similar) machines, the timestamps of all files in the /etc/logrotate.d directory appear to correspond to the time of the last boot.

On both machines, the timestamps of files in other directories are staggered as one would expect.

I don't mean to shoot down the only plausible theory, because I'm fresh out of alternatives, but both machines appear to be reading and writing to "disk" with no issues.

In fact the "other" one is right now successfully pulling down all my torrent traffic, as it has been doing for months. (I don't have another card large enough to substitute for that one.)


Crossed midnight again just now, and both machines have "refreshed" their ghosts to an Apr 1 00:00 timestamp.

I'm clearly haunted. Maybe if I say "Beetlejuice" three times out loud,...