LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - Newbie (https://www.linuxquestions.org/questions/linux-newbie-8/)
-   -   oracle11i 100% full even delete file (https://www.linuxquestions.org/questions/linux-newbie-8/oracle11i-100-full-even-delete-file-4175639551/)

macuacua 10-01-2018 09:17 PM

oracle11i 100% full even delete file
 
I am having problem with my system that make me confuse. When I run df -h I get
/dev/sda1 483G 455G 3.1G 100% /oracle11i

But when I do du -sh oracleiii I get another answer
[root@ebs3 ~]# cd /oracle11i
[root@ebs3 oracle11i]# du -sh
124G .

I did reboot of the server after deleting some files but the problem remain.

Where is going my space and how can I recover the space?

Please help me to fix this problem because I my system is down.

Regards

macuacua

berndbausch 10-01-2018 09:33 PM

Most probably, some process(es) (from Oracle, in all likelihood) opened large files, then removed their directory entries. You end up with "hidden" files - files with no names that continue to occupy space until the process closes them or exits. This is a common practice for temporary files: They are cleaned up automatically when the process that created them disappears.

df uses a different mechanism to calculate storage usage than du. df checks the filesystem's used space statistics, whereas du goes down a filesystem hierarchy, finds all files and sums up their sizes. Since du doesn't find the "hidden" files mentioned above, it reports less space used.

Such files have a "link count" (that is the number of names of a file) of 0. You can find them this way:

Code:

$ sudo lsof +L 1
COMMAND  PID    USER  FD  TYPE DEVICE SIZE/OFF NLINK  NODE NAME
upstart  885 lightdm    9w  REG  179,2      722    0 131248 /var/lib/lightdm/.cache/upstart/indicator-sound.log.1 (deleted)
upstart  885 lightdm  10w  REG  179,2      214    0 131636 /var/lib/lightdm/.cache/upstart/indicator-messages.log.1 (deleted)
smbd    15303    root    2w  REG  179,2      154    0  1072 /var/log/samba/log.smbd.1 (deleted)
smbd    15303    root    9w  REG  179,2      154    0  1072 /var/log/samba/log.smbd.1 (deleted)

NLINK is the link count.

You probably recover your space when you stop the database.

macuacua 10-01-2018 09:53 PM

Hi,

I did 2 times restart. I did database rebbot and the problem was not solve. Later, I did shutdown of database and reboot the server, even that the problem remain.
It is why I think I am looking for another procedure to fix problem.

macuacua 10-01-2018 09:57 PM

Code:

This picture show the deleted files.
[root@ebs3 ~]# /usr/sbin/lsof | grep deleted | more
f60webmx  1988    oracle  61u      REG      104,7    159766    9076806 /tmp/f
ileMTIm4P.TMP (deleted)
gpm        4157      root    0u      CHR      136,0                    2 /dev/p
ts/0 (deleted)
gpm        4157      root    1u      CHR      136,0                    2 /dev/p
ts/0 (deleted)
gpm        4157      root    2u      CHR      136,0                    2 /dev/p
ts/0 (deleted)
oracle    4769    oracle  12u      REG        8,1          0  58607526 /oracl
e11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle    4771    oracle  12u      REG        8,1          0  58607526 /oracl
e11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle    4773    oracle  12u      REG        8,1          0  58607526 /oracl
e11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle    4775    oracle  12u      REG        8,1          0  58607526 /oracl
e11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle    4777    oracle  12u      REG        8,1          0  58607526 /oracl
e11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle    4779    oracle  12u      REG        8,1          0  58607526 /oracl
e11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle    4781    oracle  12u      REG        8,1          0  58607526 /oracl
e11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle    4790    oracle  12u      REG        8,1          0  58607526 /oracl
e11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle    4808    oracle  12u      REG        8,1          0  58607526 /oracle11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle    4815    oracle  12u      REG        8,1          0  58607526 /oracle11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle    4817    oracle  12u      REG        8,1          0  58607526 /oracle11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle    4821    oracle  12u      REG        8,1          0  58607526 /oracle11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle    4823    oracle  12u      REG        8,1          0  58607526 /oracle11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle    4827    oracle  12u      REG        8,1          0  58607526 /oracle11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle    5016    oracle  12u      REG        8,1          0  58607526 /oracle11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle    5018    oracle  12u      REG        8,1          0  58607526 /oracle11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
sh        5858    oracle    0r      REG      104,7      1999    9076753 /tmp/sh-thd-1538378524 (deleted)
FNDLIBR    5863    oracle    0r      REG      104,7        19    9076754 /tmp/sh-thd-1538376526 (deleted)
f60webmx  6518    oracle  57u      REG      104,7    5329433    9076810 /tmp/filerCElCo.TMP (deleted)
gconfd-2  7669    oracle  13wW    REG      104,7        617    9076766 /tmp/gconfd-oracle/lock/0t1538392078ut614000u500p76
69r322682820k3220499064 (deleted)
f60webmx  19739    oracle  66u      REG      104,7    1683282    9076808 /tmp/fileqtXZ0V.TMP (deleted)
f60webmx  27924    oracle  58u      REG      104,7    2551634    9076815 /tmp/filebZvaBE.TMP (deleted)
[root@ebs3 ~]#

may be it can help you to see what can be done-

macuacua 10-02-2018 12:07 AM

Hi,
Code:

[root@ebs3 oracle11i]# sudo lsof +L 1
COMMAND  PID  USER  FD  TYPE DEVICE SIZE NLINK    NODE NAME
gpm      4157  root    0u  CHR  136,0          0        2 /dev/pts/0 (deleted)
gpm      4157  root    1u  CHR  136,0          0        2 /dev/pts/0 (deleted)
gpm      4157  root    2u  CHR  136,0          0        2 /dev/pts/0 (deleted)
oracle  4769 oracle  12u  REG    8,1    0    0 58607526 /oracle11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle  4771 oracle  12u  REG    8,1    0    0 58607526 /oracle11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle  4773 oracle  12u  REG    8,1    0    0 58607526 /oracle11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle  4775 oracle  12u  REG    8,1    0    0 58607526 /oracle11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle  4777 oracle  12u  REG    8,1    0    0 58607526 /oracle11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle  4779 oracle  12u  REG    8,1    0    0 58607526 /oracle11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle  4781 oracle  12u  REG    8,1    0    0 58607526 /oracle11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle  4790 oracle  12u  REG    8,1    0    0 58607526 /oracle11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle  4808 oracle  12u  REG    8,1    0    0 58607526 /oracle11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle  4815 oracle  12u  REG    8,1    0    0 58607526 /oracle11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle  4817 oracle  12u  REG    8,1    0    0 58607526 /oracle11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle  4821 oracle  12u  REG    8,1    0    0 58607526 /oracle11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle  4823 oracle  12u  REG    8,1    0    0 58607526 /oracle11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle  4827 oracle  12u  REG    8,1    0    0 58607526 /oracle11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle  5016 oracle  12u  REG    8,1    0    0 58607526 /oracle11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
sh      5858 oracle    0r  REG  104,7 1999    0  9076753 /tmp/sh-thd-1538378524 (deleted)
FNDLIBR  5863 oracle    0r  REG  104,7  19    0  9076754 /tmp/sh-thd-1538376526 (deleted)
oracle  11545 oracle  12u  REG    8,1    0    0 58607526 /oracle11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle  13107 oracle  12u  REG    8,1    0    0 58607526 /oracle11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
[root@ebs3 oracle11i]#


macuacua 10-02-2018 12:27 AM

Hi,

Someone have answer for this issue?

macuacua

berndbausch 10-02-2018 01:18 AM

Can you put [code][/code] tags around your output? It is hard to read without any formatting.

macuacua 10-02-2018 02:49 AM

Mr
 
1 Attachment(s)
Hi Ben,

I upload one file

macuacua 10-02-2018 02:59 AM

Mr
 
1 Attachment(s)
Hi,

I upload in text file. If is not in readable, please advise.

Thank

Macuacua

berndbausch 10-02-2018 04:21 AM

I see a 2GB file, which is large, but nothing that explains your "loss" of about 300GB. So, right now I wonder what that might be.

Can you start the server without Oracle and check again? Also run a file system check on /dev/sda1.

What type of filesystem is that? I would like to see the superblock; perhaps it contains clues.

By the way, in this forum you can edit what you wrote (there is an Edit button), then add [code] before the output, and [/code] after it, so that it is formatted in a readable way.

macuacua 10-02-2018 04:42 AM

Hi Beni,

In fact it is strange issue that I have.
The disks is come from netapp storage and I checked for an snapshot or something else, nothing.
Yesterday I did reboot server and poweroff (remove power cable) and I will repeat this procedure after one hour.

Did you identify which PID I can kill to get back those 2G?
On that filesystem there are oracle database.

Thanks

TB0ne 10-02-2018 06:56 AM

Quote:

Originally Posted by macuacua (Post 5910084)
Hi Beni,
In fact it is strange issue that I have. The disks is come from netapp storage and I checked for an snapshot or something else, nothing. Yesterday I did reboot server and poweroff (remove power cable) and I will repeat this procedure after one hour. Did you identify which PID I can kill to get back those 2G? On that filesystem there are oracle database.

Nothing strange about this at all. Oracle reserves disk space for things, and rebooting/deleting won't necessarily help. What does your DBA say about this? Simple way to check things is to bring up your server with Oracle shut down, and see what happens to your disk space.

And have you contacted Oracle?? Oracle 11i is a commercial, pay-for product, and you get support when you pay. They can assist you with this. Also, as said, use CODE tags when posting things (don't put pictures or upload huge files), and don't bump your own thread after a short time, asking for help. If you're in a hurry, use the paid-for support from Oracle.

macuacua 10-02-2018 07:06 AM

Hi,

What can I did shotdown now is bad

SQL> conn /as sysdba
ERROR:
ORA-09925: Unable to create audit trail file
Linux Error: 30: Read-only file system
Additional information: 9925
ORA-09925: Unable to create audit trail file
Linux Error: 30: Read-only file system
Additional information: 9925

URGENT

berndbausch 10-02-2018 07:18 AM

Quote:

Linux Error: 30: Read-only file system
This looks like you somehow mounted your filesystem read-only. What does the mount command say?

DISCLAIMER: I know a little bit about Linux, but close to nothing about Oracle.

TB0ne 10-02-2018 07:47 AM

Quote:

Originally Posted by macuacua (Post 5910122)
Hi,

What can I did shotdown now is bad

SQL> conn /as sysdba
ERROR:
ORA-09925: Unable to create audit trail file
Linux Error: 30: Read-only file system
Additional information: 9925
ORA-09925: Unable to create audit trail file
Linux Error: 30: Read-only file system
Additional information: 9925

URGENT

You can first read the "How to ask a smart question" link...this is NOT URGENT for anyone here. We volunteer our time, so asking for/expecting 'urgent' help is plain rude. Secondly, you were given pieces of advice:
  • Restart the system WITHOUT ORACLE RUNNING, to see what happens to your disk space
  • Ask your DBA for assistance
  • CALL ORACLE SUPPORT
Have you done any of these things?? Your Oracle DB is down, because it is either misconfigured, running on a partition that's too small, or both. AGAIN: get your DBA to look at how Oracle is configured, and what it's doing. Call Oracle support, because **AGAIN** Oracle is not free. Oracle won't run when it's out of disk space, anymore than you can log in to your Linux system when the disk is 100% full.

Oracle's knowledgebase has many articles about this. Have you run disk cleanup?
https://blog.rackspace.com/just-the-...cle-disk-space


All times are GMT -5. The time now is 10:14 PM.