LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 03-28-2016, 04:34 AM   #1
bosong
LQ Newbie
 
Registered: Mar 2016
Posts: 16

Rep: Reputation: Disabled
sorting


I need to sort a massive file, for username and command used, is there any simple way?
 
Old 03-28-2016, 04:38 AM   #2
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 20,181

Rep: Reputation: 3756Reputation: 3756Reputation: 3756Reputation: 3756Reputation: 3756Reputation: 3756Reputation: 3756Reputation: 3756Reputation: 3756Reputation: 3756Reputation: 3756
"man sort" would be a good start.
 
2 members found this post helpful.
Old 03-28-2016, 04:52 AM   #3
bosong
LQ Newbie
 
Registered: Mar 2016
Posts: 16

Original Poster
Rep: Reputation: Disabled
how do i sort out and display only the 3rd column?? is it possible?
 
Old 03-28-2016, 05:12 AM   #4
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 20,181

Rep: Reputation: 3756Reputation: 3756Reputation: 3756Reputation: 3756Reputation: 3756Reputation: 3756Reputation: 3756Reputation: 3756Reputation: 3756Reputation: 3756Reputation: 3756
Read the doco, do some testing, see what happens. That's how you learn.
Use a subset of the data - 10-20 records should do.
 
Old 03-28-2016, 09:46 AM   #5
sundialsvcs
LQ Guru
 
Registered: Feb 2004
Location: SE Tennessee, USA
Distribution: Gentoo, LFS
Posts: 9,542
Blog Entries: 4

Rep: Reputation: 3433Reputation: 3433Reputation: 3433Reputation: 3433Reputation: 3433Reputation: 3433Reputation: 3433Reputation: 3433Reputation: 3433Reputation: 3433Reputation: 3433
Believe it or not, the sort command can do things like that.

I don't know what you consider "massive." (What's the order of magnitude? MIllions of records? Tens of millions? Hundreds? Only the latter category might be considered truly "massive," and only in the sense that it will take a little while longer.)
 
Old 03-28-2016, 10:24 AM   #6
dijetlo
Senior Member
 
Registered: Jan 2009
Location: RHELtopia....
Distribution: Solaris 11.2/Slackware/RHEL/
Posts: 1,491
Blog Entries: 2

Rep: Reputation: Disabled
Check the size of /tmp since when your massive file gets sorted, that's where the scratch files get created.
 
Old 03-28-2016, 10:27 AM   #7
BW-userx
LQ Guru
 
Registered: Sep 2013
Location: Somewhere in my head.
Distribution: Slackware (current), FreeBSD, Win10, It varies
Posts: 9,952

Rep: Reputation: 2149Reputation: 2149Reputation: 2149Reputation: 2149Reputation: 2149Reputation: 2149Reputation: 2149Reputation: 2149Reputation: 2149Reputation: 2149Reputation: 2149
massive: exceptionally large

exceptionally: to a greater degree than normal; unusually
large: of considerable or relatively great size, extent, or capacity

by his usage of the term massive file would then indicate this is is abnormally or unusually large.
 
Old 03-28-2016, 10:13 PM   #8
bosong
LQ Newbie
 
Registered: Mar 2016
Posts: 16

Original Poster
Rep: Reputation: Disabled
The server I am doing has maybe about 1 million records per day, i want to sort out the username along side with the command use and also count for each command, is there anyway i can do it???
 
Old 03-28-2016, 10:58 PM   #9
bosong
LQ Newbie
 
Registered: Mar 2016
Posts: 16

Original Poster
Rep: Reputation: Disabled
There are about 100 users and thousands of different commands
Can anyone help me?
Thanks in advance
 
Old 03-29-2016, 04:31 AM   #10
tshikose
Member
 
Registered: Apr 2010
Location: Kinshasa, Democratic Republic of Congo
Distribution: RHEL, Fedora, CentOS
Posts: 523

Rep: Reputation: 95
Hi,

You have been giving good advices to start with the sort command.
Apparently, you are asking for more, without sharing here what you have tried so far.

Remember that while people are keen to help, they do not want to do your tasks or your homework.
Read the man page of sort, and test something.
 
Old 03-29-2016, 06:36 AM   #11
pan64
LQ Addict
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 17,567

Rep: Reputation: 5949Reputation: 5949Reputation: 5949Reputation: 5949Reputation: 5949Reputation: 5949Reputation: 5949Reputation: 5949Reputation: 5949Reputation: 5949Reputation: 5949
Yes, you have got help already: see man sort, and if you need examples just look for tutorials/examples (of sort) on the web. You will find everything you need. In case you want to fine-tune a solution just show us what you have already picked and what is your problem with that. Since you gave no usable info we can hardly give you usable answer (by the way the answer is yes, there is a simple way)
https://is.gd/8WT0AM
 
Old 03-29-2016, 08:20 AM   #12
rtmistler
Moderator
 
Registered: Mar 2011
Location: USA
Distribution: MINT Debian, Angstrom, SUSE, Ubuntu, Debian
Posts: 9,461
Blog Entries: 13

Rep: Reputation: 4567Reputation: 4567Reputation: 4567Reputation: 4567Reputation: 4567Reputation: 4567Reputation: 4567Reputation: 4567Reputation: 4567Reputation: 4567Reputation: 4567
@OP Welcome to LQ. Please review that link.

Recommend you attempt the sort command and then report how that attempt occurred.

Another thought is that in addition to sorting, you can also reduce this massive data set by using grep to only locate the records which matter out of that very large file.

When reporting results, it is best to place them within [code] tags.
 
Old 03-29-2016, 08:31 AM   #13
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,912

Rep: Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513
If you are using a system with /tmp as tmpfs, you will likely need to use the
-T option to use an alternate location for your scratch files while sorting.

There is a hint that using an environment variable TMPDIR may redirect it also.
 
Old 03-30-2016, 05:35 AM   #14
bosong
LQ Newbie
 
Registered: Mar 2016
Posts: 16

Original Poster
Rep: Reputation: Disabled
Sorry for the late reply.
Thanks all for help
I have solve the problem through using sort + awk + uniq
 
Old 03-30-2016, 08:10 AM   #15
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,912

Rep: Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513
Those are frequently the solution. Sometimes the uniq is extra as sort has a unique option.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Rapid Listing, alphabetically sorting, dir/files sorting in C ? Xeratul Programming 18 11-24-2014 11:13 AM
sorting mohamad Linux - Newbie 7 06-23-2010 07:46 AM
Java sorting manolakis Programming 2 07-21-2008 12:28 PM
Sorting Beppe83 Linux - Software 7 06-21-2004 10:10 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 11:08 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration