LinuxQuestions.org
Visit Jeremy's Blog.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 10-01-2018, 09:17 PM   #1
macuacua
LQ Newbie
 
Registered: Oct 2018
Posts: 11

Rep: Reputation: Disabled
Angry oracle11i 100% full even delete file


I am having problem with my system that make me confuse. When I run df -h I get
/dev/sda1 483G 455G 3.1G 100% /oracle11i

But when I do du -sh oracleiii I get another answer
[root@ebs3 ~]# cd /oracle11i
[root@ebs3 oracle11i]# du -sh
124G .

I did reboot of the server after deleting some files but the problem remain.

Where is going my space and how can I recover the space?

Please help me to fix this problem because I my system is down.

Regards

macuacua

Last edited by macuacua; 10-01-2018 at 09:41 PM. Reason: more details
 
Old 10-01-2018, 09:33 PM   #2
berndbausch
LQ Addict
 
Registered: Nov 2013
Location: Tokyo
Distribution: Mostly Ubuntu and Centos
Posts: 6,316

Rep: Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002
Most probably, some process(es) (from Oracle, in all likelihood) opened large files, then removed their directory entries. You end up with "hidden" files - files with no names that continue to occupy space until the process closes them or exits. This is a common practice for temporary files: They are cleaned up automatically when the process that created them disappears.

df uses a different mechanism to calculate storage usage than du. df checks the filesystem's used space statistics, whereas du goes down a filesystem hierarchy, finds all files and sums up their sizes. Since du doesn't find the "hidden" files mentioned above, it reports less space used.

Such files have a "link count" (that is the number of names of a file) of 0. You can find them this way:

Code:
$ sudo lsof +L 1
COMMAND   PID    USER   FD   TYPE DEVICE SIZE/OFF NLINK   NODE NAME
upstart   885 lightdm    9w   REG  179,2      722     0 131248 /var/lib/lightdm/.cache/upstart/indicator-sound.log.1 (deleted)
upstart   885 lightdm   10w   REG  179,2      214     0 131636 /var/lib/lightdm/.cache/upstart/indicator-messages.log.1 (deleted)
smbd    15303    root    2w   REG  179,2      154     0   1072 /var/log/samba/log.smbd.1 (deleted)
smbd    15303    root    9w   REG  179,2      154     0   1072 /var/log/samba/log.smbd.1 (deleted)
NLINK is the link count.

You probably recover your space when you stop the database.

Last edited by berndbausch; 10-01-2018 at 09:38 PM. Reason: added lsof
 
Old 10-01-2018, 09:53 PM   #3
macuacua
LQ Newbie
 
Registered: Oct 2018
Posts: 11

Original Poster
Rep: Reputation: Disabled
Hi,

I did 2 times restart. I did database rebbot and the problem was not solve. Later, I did shutdown of database and reboot the server, even that the problem remain.
It is why I think I am looking for another procedure to fix problem.
 
Old 10-01-2018, 09:57 PM   #4
macuacua
LQ Newbie
 
Registered: Oct 2018
Posts: 11

Original Poster
Rep: Reputation: Disabled
Code:
This picture show the deleted files.
[root@ebs3 ~]# /usr/sbin/lsof | grep deleted | more
f60webmx   1988    oracle   61u      REG      104,7     159766    9076806 /tmp/f
ileMTIm4P.TMP (deleted)
gpm        4157      root    0u      CHR      136,0                     2 /dev/p
ts/0 (deleted)
gpm        4157      root    1u      CHR      136,0                     2 /dev/p
ts/0 (deleted)
gpm        4157      root    2u      CHR      136,0                     2 /dev/p
ts/0 (deleted)
oracle     4769    oracle   12u      REG        8,1          0   58607526 /oracl
e11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle     4771    oracle   12u      REG        8,1          0   58607526 /oracl
e11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle     4773    oracle   12u      REG        8,1          0   58607526 /oracl
e11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle     4775    oracle   12u      REG        8,1          0   58607526 /oracl
e11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle     4777    oracle   12u      REG        8,1          0   58607526 /oracl
e11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle     4779    oracle   12u      REG        8,1          0   58607526 /oracl
e11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle     4781    oracle   12u      REG        8,1          0   58607526 /oracl
e11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle     4790    oracle   12u      REG        8,1          0   58607526 /oracl
e11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle     4808    oracle   12u      REG        8,1          0   58607526 /oracle11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle     4815    oracle   12u      REG        8,1          0   58607526 /oracle11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle     4817    oracle   12u      REG        8,1          0   58607526 /oracle11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle     4821    oracle   12u      REG        8,1          0   58607526 /oracle11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle     4823    oracle   12u      REG        8,1          0   58607526 /oracle11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle     4827    oracle   12u      REG        8,1          0   58607526 /oracle11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle     5016    oracle   12u      REG        8,1          0   58607526 /oracle11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle     5018    oracle   12u      REG        8,1          0   58607526 /oracle11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
sh         5858    oracle    0r      REG      104,7       1999    9076753 /tmp/sh-thd-1538378524 (deleted)
FNDLIBR    5863    oracle    0r      REG      104,7         19    9076754 /tmp/sh-thd-1538376526 (deleted)
f60webmx   6518    oracle   57u      REG      104,7    5329433    9076810 /tmp/filerCElCo.TMP (deleted)
gconfd-2   7669    oracle   13wW     REG      104,7        617    9076766 /tmp/gconfd-oracle/lock/0t1538392078ut614000u500p76
69r322682820k3220499064 (deleted)
f60webmx  19739    oracle   66u      REG      104,7    1683282    9076808 /tmp/fileqtXZ0V.TMP (deleted)
f60webmx  27924    oracle   58u      REG      104,7    2551634    9076815 /tmp/filebZvaBE.TMP (deleted)
[root@ebs3 ~]#
may be it can help you to see what can be done-

Last edited by macuacua; 10-02-2018 at 04:31 AM.
 
Old 10-02-2018, 12:07 AM   #5
macuacua
LQ Newbie
 
Registered: Oct 2018
Posts: 11

Original Poster
Rep: Reputation: Disabled
Angry

Hi,
Code:
[root@ebs3 oracle11i]# sudo lsof +L 1
COMMAND   PID   USER   FD   TYPE DEVICE SIZE NLINK     NODE NAME
gpm      4157   root    0u   CHR  136,0          0        2 /dev/pts/0 (deleted)
gpm      4157   root    1u   CHR  136,0          0        2 /dev/pts/0 (deleted)
gpm      4157   root    2u   CHR  136,0          0        2 /dev/pts/0 (deleted)
oracle   4769 oracle   12u   REG    8,1    0     0 58607526 /oracle11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle   4771 oracle   12u   REG    8,1    0     0 58607526 /oracle11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle   4773 oracle   12u   REG    8,1    0     0 58607526 /oracle11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle   4775 oracle   12u   REG    8,1    0     0 58607526 /oracle11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle   4777 oracle   12u   REG    8,1    0     0 58607526 /oracle11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle   4779 oracle   12u   REG    8,1    0     0 58607526 /oracle11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle   4781 oracle   12u   REG    8,1    0     0 58607526 /oracle11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle   4790 oracle   12u   REG    8,1    0     0 58607526 /oracle11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle   4808 oracle   12u   REG    8,1    0     0 58607526 /oracle11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle   4815 oracle   12u   REG    8,1    0     0 58607526 /oracle11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle   4817 oracle   12u   REG    8,1    0     0 58607526 /oracle11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle   4821 oracle   12u   REG    8,1    0     0 58607526 /oracle11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle   4823 oracle   12u   REG    8,1    0     0 58607526 /oracle11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle   4827 oracle   12u   REG    8,1    0     0 58607526 /oracle11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle   5016 oracle   12u   REG    8,1    0     0 58607526 /oracle11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
sh       5858 oracle    0r   REG  104,7 1999     0  9076753 /tmp/sh-thd-1538378524 (deleted)
FNDLIBR  5863 oracle    0r   REG  104,7   19     0  9076754 /tmp/sh-thd-1538376526 (deleted)
oracle  11545 oracle   12u   REG    8,1    0     0 58607526 /oracle11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
oracle  13107 oracle   12u   REG    8,1    0     0 58607526 /oracle11i/proddb/10.2.0/dbs/lkinstPROD (deleted)
[root@ebs3 oracle11i]#

Last edited by macuacua; 10-02-2018 at 04:37 AM.
 
Old 10-02-2018, 12:27 AM   #6
macuacua
LQ Newbie
 
Registered: Oct 2018
Posts: 11

Original Poster
Rep: Reputation: Disabled
Hi,

Someone have answer for this issue?

macuacua
 
Old 10-02-2018, 01:18 AM   #7
berndbausch
LQ Addict
 
Registered: Nov 2013
Location: Tokyo
Distribution: Mostly Ubuntu and Centos
Posts: 6,316

Rep: Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002
Can you put [code][/code] tags around your output? It is hard to read without any formatting.
 
Old 10-02-2018, 02:49 AM   #8
macuacua
LQ Newbie
 
Registered: Oct 2018
Posts: 11

Original Poster
Rep: Reputation: Disabled
Angry Mr

Hi Ben,

I upload one file
Attached Thumbnails
Click image for larger version

Name:	macuacua1.png
Views:	16
Size:	129.6 KB
ID:	28694  
 
Old 10-02-2018, 02:59 AM   #9
macuacua
LQ Newbie
 
Registered: Oct 2018
Posts: 11

Original Poster
Rep: Reputation: Disabled
Angry Mr

Hi,

I upload in text file. If is not in readable, please advise.

Thank

Macuacua
Attached Files
File Type: txt macuacua.txt (9.9 KB, 9 views)
 
Old 10-02-2018, 04:21 AM   #10
berndbausch
LQ Addict
 
Registered: Nov 2013
Location: Tokyo
Distribution: Mostly Ubuntu and Centos
Posts: 6,316

Rep: Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002
I see a 2GB file, which is large, but nothing that explains your "loss" of about 300GB. So, right now I wonder what that might be.

Can you start the server without Oracle and check again? Also run a file system check on /dev/sda1.

What type of filesystem is that? I would like to see the superblock; perhaps it contains clues.

By the way, in this forum you can edit what you wrote (there is an Edit button), then add [code] before the output, and [/code] after it, so that it is formatted in a readable way.

Last edited by berndbausch; 10-02-2018 at 04:26 AM.
 
Old 10-02-2018, 04:42 AM   #11
macuacua
LQ Newbie
 
Registered: Oct 2018
Posts: 11

Original Poster
Rep: Reputation: Disabled
Hi Beni,

In fact it is strange issue that I have.
The disks is come from netapp storage and I checked for an snapshot or something else, nothing.
Yesterday I did reboot server and poweroff (remove power cable) and I will repeat this procedure after one hour.

Did you identify which PID I can kill to get back those 2G?
On that filesystem there are oracle database.

Thanks
 
Old 10-02-2018, 06:56 AM   #12
TB0ne
LQ Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 26,632

Rep: Reputation: 7965Reputation: 7965Reputation: 7965Reputation: 7965Reputation: 7965Reputation: 7965Reputation: 7965Reputation: 7965Reputation: 7965Reputation: 7965Reputation: 7965
Quote:
Originally Posted by macuacua View Post
Hi Beni,
In fact it is strange issue that I have. The disks is come from netapp storage and I checked for an snapshot or something else, nothing. Yesterday I did reboot server and poweroff (remove power cable) and I will repeat this procedure after one hour. Did you identify which PID I can kill to get back those 2G? On that filesystem there are oracle database.
Nothing strange about this at all. Oracle reserves disk space for things, and rebooting/deleting won't necessarily help. What does your DBA say about this? Simple way to check things is to bring up your server with Oracle shut down, and see what happens to your disk space.

And have you contacted Oracle?? Oracle 11i is a commercial, pay-for product, and you get support when you pay. They can assist you with this. Also, as said, use CODE tags when posting things (don't put pictures or upload huge files), and don't bump your own thread after a short time, asking for help. If you're in a hurry, use the paid-for support from Oracle.
 
Old 10-02-2018, 07:06 AM   #13
macuacua
LQ Newbie
 
Registered: Oct 2018
Posts: 11

Original Poster
Rep: Reputation: Disabled
Hi,

What can I did shotdown now is bad

SQL> conn /as sysdba
ERROR:
ORA-09925: Unable to create audit trail file
Linux Error: 30: Read-only file system
Additional information: 9925
ORA-09925: Unable to create audit trail file
Linux Error: 30: Read-only file system
Additional information: 9925

URGENT
 
Old 10-02-2018, 07:18 AM   #14
berndbausch
LQ Addict
 
Registered: Nov 2013
Location: Tokyo
Distribution: Mostly Ubuntu and Centos
Posts: 6,316

Rep: Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002
Quote:
Linux Error: 30: Read-only file system
This looks like you somehow mounted your filesystem read-only. What does the mount command say?

DISCLAIMER: I know a little bit about Linux, but close to nothing about Oracle.
 
Old 10-02-2018, 07:47 AM   #15
TB0ne
LQ Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 26,632

Rep: Reputation: 7965Reputation: 7965Reputation: 7965Reputation: 7965Reputation: 7965Reputation: 7965Reputation: 7965Reputation: 7965Reputation: 7965Reputation: 7965Reputation: 7965
Quote:
Originally Posted by macuacua View Post
Hi,

What can I did shotdown now is bad

SQL> conn /as sysdba
ERROR:
ORA-09925: Unable to create audit trail file
Linux Error: 30: Read-only file system
Additional information: 9925
ORA-09925: Unable to create audit trail file
Linux Error: 30: Read-only file system
Additional information: 9925

URGENT
You can first read the "How to ask a smart question" link...this is NOT URGENT for anyone here. We volunteer our time, so asking for/expecting 'urgent' help is plain rude. Secondly, you were given pieces of advice:
  • Restart the system WITHOUT ORACLE RUNNING, to see what happens to your disk space
  • Ask your DBA for assistance
  • CALL ORACLE SUPPORT
Have you done any of these things?? Your Oracle DB is down, because it is either misconfigured, running on a partition that's too small, or both. AGAIN: get your DBA to look at how Oracle is configured, and what it's doing. Call Oracle support, because **AGAIN** Oracle is not free. Oracle won't run when it's out of disk space, anymore than you can log in to your Linux system when the disk is 100% full.

Oracle's knowledgebase has many articles about this. Have you run disk cleanup?
https://blog.rackspace.com/just-the-...cle-disk-space
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Root file system / 100 % full moata_u Solaris / OpenSolaris 15 06-15-2011 10:12 AM
[SOLVED] System File is Full (100%) How can I relieve the situation? mike-member Linux - Newbie 8 03-14-2011 02:32 AM
/home file system 100% full gromo Linux - Enterprise 5 11-04-2004 09:47 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 06:39 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration