LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 08-22-2022, 07:32 PM   #1
ericlindellnyc
Member
 
Registered: Jun 2017
Posts: 132

Rep: Reputation: Disabled
How to Conditionally Execute Code If File in OS-Walk is Empty in Python MacOS


Code:
import os
for root, dirs, files in os.walk("/Volumes/TM3gbBu/miscx4now/fr6tb24jul22/miscxDox-ddupt-8gb/sort/BUs/doxx"):
    for name in files:
        print('filename is ', name)
        fileSize = os.path.getsize(name)
        print('file size is ', fileSize)
. . produces this result . .

Code:
 Traceback (most recent call last):
  File "testOsWalk.py", line 5, in <module>
    fileSize = os.path.getsize(name)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/genericpath.py", line 50, in getsize
    return os.stat(filename).st_size
FileNotFoundError: [Errno 2] No such file or directory: '._.DS_Store'
If I try to determine fileSize with
Code:
fileSize = os.path.getsize(name)
it tells me "No such file or directory"
But it's referring to a file or directory that's THERE.

If I omit the two lines about file size and just include the line before that to print file name,
it prints out all files in the directory, as expected.

In another version of this script, it would print out file size of hidden file but not visible file.

Confused. Help much appreciated.

Last edited by ericlindellnyc; 08-22-2022 at 09:20 PM. Reason: to clarify
 
Old 08-22-2022, 08:03 PM   #2
dugan
LQ Guru
 
Registered: Nov 2003
Location: Canada
Distribution: distro hopper
Posts: 10,636

Rep: Reputation: 5024Reputation: 5024Reputation: 5024Reputation: 5024Reputation: 5024Reputation: 5024Reputation: 5024Reputation: 5024Reputation: 5024Reputation: 5024Reputation: 5024
As is typical, I'm going to start by being that guy. You've confirmed that it's not a broken link? What do os.path.exists(), os.path.isfile(), and similar things return?

Last edited by dugan; 08-22-2022 at 08:07 PM.
 
Old 08-22-2022, 08:21 PM   #3
michaelk
Moderator
 
Registered: Aug 2002
Posts: 23,481

Rep: Reputation: 5091Reputation: 5091Reputation: 5091Reputation: 5091Reputation: 5091Reputation: 5091Reputation: 5091Reputation: 5091Reputation: 5091Reputation: 5091Reputation: 5091
Code:
for root, dirs, files in os.walk("/Volumes/TM3gbBu/miscx4now/fr6tb24jul22/miscxDox-ddupt-8gb/sort/BUs/doxx", topdown=True):
   for name in files:
      print(os.path.join(root, name))
      filename=os.path.join(root,name)
      fileSize = os.path.getsize(filename)
      print('file size is ', fileSize)
files does not include the full path.

Last edited by michaelk; 08-22-2022 at 08:46 PM.
 
Old 08-22-2022, 11:20 PM   #4
ericlindellnyc
Member
 
Registered: Jun 2017
Posts: 132

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by michaelk View Post
Code:
for root, dirs, files in os.walk("/Volumes/TM3gbBu/miscx4now/fr6tb24jul22/miscxDox-ddupt-8gb/sort/BUs/doxx", topdown=True):
   for name in files:
      print(os.path.join(root, name))
      filename=os.path.join(root,name)
      fileSize = os.path.getsize(filename)
      print('file size is ', fileSize)
files does not include the full path.
Is
Code:
os.path.join(root,name)
considered the full path?
If so, I could write
Code:
fileSize = os.path.getsize(os.path.join(root,name))
but that produces the same error.
I'm not sure how else I can specify the full path when I'm doing a traversal of a big directory structure.
CORRECTION . .
Code:
fileSize = os.path.getsize(os.path.join(root,name))
DOES NOT PRODUCE THE SAME ERROR.
CODE AND OUTPUT GIVEN BELOW.
THANKS AGAIN

Last edited by ericlindellnyc; 08-23-2022 at 12:32 AM. Reason: to clarify
 
Old 08-22-2022, 11:52 PM   #5
dugan
LQ Guru
 
Registered: Nov 2003
Location: Canada
Distribution: distro hopper
Posts: 10,636

Rep: Reputation: 5024Reputation: 5024Reputation: 5024Reputation: 5024Reputation: 5024Reputation: 5024Reputation: 5024Reputation: 5024Reputation: 5024Reputation: 5024Reputation: 5024
I assumed that the code sample was completely outdated and I just ignored it. The updates (including in michaelk's edited post) look right to me.

That said, the error message you posted doesn't have a full path:

Quote:
Code:
FileNotFoundError: [Errno 2] No such file or directory: '._.DS_Store'
If you were building the paths correctly, then there would be a full path in the crash message:
Code:
>>> import os
>>> os.path.getsize('/path/does/not/exist.txt')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python3.10/genericpath.py", line 50, in getsize
    return os.stat(filename).st_size
FileNotFoundError: [Errno 2] No such file or directory: '/path/does/not/exist.txt'

Last edited by dugan; 08-23-2022 at 12:03 AM.
 
Old 08-23-2022, 12:01 AM   #6
dugan
LQ Guru
 
Registered: Nov 2003
Location: Canada
Distribution: distro hopper
Posts: 10,636

Rep: Reputation: 5024Reputation: 5024Reputation: 5024Reputation: 5024Reputation: 5024Reputation: 5024Reputation: 5024Reputation: 5024Reputation: 5024Reputation: 5024Reputation: 5024
Here's a complete demonstration that gets the sizes of two zero-length files. One hidden and one not:

Code:
❯ tree -a
.
└── root
    ├── .hidden
    └── nothidden

1 directory, 2 files

❯ python3
Python 3.10.6 (main, Aug  2 2022, 00:00:00) [GCC 12.1.1 20220507 (Red Hat 12.1.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> for root, _, files in os.walk('./root'):
...     for name in files:
...         path = os.path.join(root, name)
...         print(path, os.path.getsize(path))
... 
./root/.hidden 0
./root/nothidden 0

Last edited by dugan; 08-23-2022 at 12:02 AM.
 
Old 08-23-2022, 12:27 AM   #7
ericlindellnyc
Member
 
Registered: Jun 2017
Posts: 132

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by dugan View Post
As is typical, I'm going to start by being that guy. You've confirmed that it's not a broken link? What do os.path.exists(), os.path.isfile(), and similar things return?
It wasn't a broken link. It was throwing the error for a file that definitely wasn't a link.
That said, there were broken links in the traversed folder, which I have removed.

Your suggestion was very fruitful, implemented as follows.
import os
for root, dirs, files in os.walk("/Volumes/TM3gbBu/miscx4now/fr6tb24jul22/miscxDox-ddupt-8gb/sort/BUs/doxx"):
for name in files:
print('filename is ', name)

if os.path.exists(name):
print('it exists')
else:
print('it doesnt exist')
if os.path.isfile(name):
print('its a file')
else:
print('its not a file')


fileSize = os.path.getsize(os.path.join(root,name))
# fileSize = os.path.getsize(name)
print('file size is ', fileSize)
producing this result for three files (1st hidden; 2nd this script; 3rd dat file)

Code:
filename is  .DS_Store
it exists
its a file
file size is  6148
filename is  testOsWalk.py
it exists
its a file
file size is  559
filename is  1393748937451_AUTO.dat
it doesnt exist
its not a file
file size is  27072
There's another "dat" file that produces the same nonsensical result (saying it doesn't exist and printing its size).
But this works on the other two.
Hooray !! thanks much.
I will continue troubleshooting

Last edited by ericlindellnyc; 08-23-2022 at 12:29 AM. Reason: to clarify
 
Old 08-23-2022, 12:44 AM   #8
dugan
LQ Guru
 
Registered: Nov 2003
Location: Canada
Distribution: distro hopper
Posts: 10,636

Rep: Reputation: 5024Reputation: 5024Reputation: 5024Reputation: 5024Reputation: 5024Reputation: 5024Reputation: 5024Reputation: 5024Reputation: 5024Reputation: 5024Reputation: 5024
Quote:
On some platforms, this function [os.path.exists] may return False if permission is not granted to execute os.stat() on the requested file, even if the path physically exists.
https://docs.python.org/dev/library/...os.path.exists

Can you run "stat" on the file in a terminal?

Last edited by dugan; 08-23-2022 at 12:45 AM.
 
Old 08-23-2022, 08:55 AM   #9
michaelk
Moderator
 
Registered: Aug 2002
Posts: 23,481

Rep: Reputation: 5091Reputation: 5091Reputation: 5091Reputation: 5091Reputation: 5091Reputation: 5091Reputation: 5091Reputation: 5091Reputation: 5091Reputation: 5091Reputation: 5091
Code:
if os.path.exists(name):
  print('it exists')
else:
  print('it doesnt exist')
if os.path.isfile(name):
  print('its a file')
else:
  print('its not a file')

fileSize = os.path.getsize(os.path.join(root,name))
For simplicity lets call path1=/Volumes/TM3gbBu/miscx4now/fr6tb24jul22/miscxDox-ddupt-8gb/sort/BUs/doxx

Without knowing the exact file structure of path1 or where 1393748937451_AUTO.dat is located I will assume it is located in a subdirectory of path1.

os.path.exists(name) will only be true if testOsWalk.py is in the same directory as the file because name does not include the full path.

if 1393748937451_AUTO.dat is actually located in /path1/dir/1393748937451_AUTO.dat the above test fails.

fileSize = os.path.getsize(os.path.join(root,name)) works regardless because it is using the full path to 1393748937451_AUTO.dat i.e /path1/dir1/1393748937451_AUTO.dat.
 
  


Reply

Tags
python


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
LXer: How To Empty a File, Delete N Lines From a File, Remove Matching String From a File, And Remove Empty/Blank Lines From a File In Linux LXer Syndicated Linux News 0 11-22-2017 12:30 PM
LXer: OpenForum Europe Challenges Governments to Walk the Open Format Walk LXer Syndicated Linux News 0 09-15-2014 03:41 PM
python os.walk file search question (un expected duplicate thread) bloodyscript Programming 2 09-26-2011 10:52 AM
python os.walk file search question [SOLVED] bloodyscript Programming 3 09-24-2011 05:09 PM
conditionally deleting lines from file vaibhav_soham Linux - General 1 03-10-2011 05:45 AM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 04:52 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration