LinuxQuestions.org
LinuxAnswers - the LQ Linux tutorial section.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Networking
User Name
Password
Linux - Networking This forum is for any issue related to networks or networking.
Routing, network cards, OSI, etc. Anything is fair game.

Notices

Reply
 
LinkBack Search this Thread
Old 03-18-2013, 12:45 AM   #1
your_shadow03
Senior Member
 
Registered: Jun 2008
Location: Germany
Distribution: Slackware
Posts: 1,429
Blog Entries: 6

Rep: Reputation: 51
Need help on parallel filsystem like lustre..


Hi,

I am trying to run Apache Hadoop project on parallel filesystem like lustre. I have 1 MDS, 2 OSS/OST and 1 Lustre Client.

My lustre client shows:
Code:
[root@lustreclient1 ~]# lfs df -h
UUID                       bytes        Used   Available Use% Mounted on
lustre-MDT0000_UUID         4.5G      274.3M        3.9G   6% /mnt/lustre[MDT:0]
lustre-OST0000_UUID         5.9G      276.1M        5.3G   5% /mnt/lustre[OST:0]
lustre-OST0001_UUID         5.9G      276.1M        5.3G   5% /mnt/lustre[OST:1]
lustre-OST0002_UUID         5.9G      276.1M        5.3G   5% /mnt/lustre[OST:2]
lustre-OST0003_UUID         5.9G      276.1M        5.3G   5% /mnt/lustre[OST:3]
lustre-OST0004_UUID         5.9G      276.1M        5.3G   5% /mnt/lustre[OST:4]
lustre-OST0005_UUID         5.9G      276.1M        5.3G   5% /mnt/lustre[OST:5]
lustre-OST0006_UUID         5.9G      276.1M        5.3G   5% /mnt/lustre[OST:6]
lustre-OST0007_UUID         5.9G      276.1M        5.3G   5% /mnt/lustre[OST:7]
lustre-OST0008_UUID         5.9G      276.1M        5.3G   5% /mnt/lustre[OST:8]
lustre-OST0009_UUID         5.9G      276.1M        5.3G   5% /mnt/lustre[OST:9]
lustre-OST000a_UUID         5.9G      276.1M        5.3G   5% /mnt/lustre[OST:10]
lustre-OST000b_UUID         5.9G      276.1M        5.3G   5% /mnt/lustre[OST:11]

filesystem summary:        70.9G        3.2G       64.0G   5% /mnt/lustre
As I was unsure about which machine I need to install Hadoop softwareon, I decided to go ahead with installing Hadoop on LustreClient1.

I configured LustreClient1 with JAVA_HOME and HADOOP parameter with the following files entry:
File: conf/core-site.xml
Code:
<property>
<name>fs.default.name</name>
<value>file:///mnt/lustre</value>
</property>
<property>
<name>mapred.system.dir</name>
<value>${fs.default.name}/hadoop_tmp/mapred/system</value>
<description>The shared directory where MapReduce stores control
files.
</description>
</property>
I dint make changes in mapred-site.xml.

Now when I start 'bin/start-mapred.sh' which tried to ssh to my own local machine. I am not sure if I am doing right.

Doubt> Do I need to have two Lustre Client for this to work?

Then I tried running wordcount program shown below:

Code:
 bin/hadoop jar hadoop-examples-1.1.1.jar wordcount /tmp/rahul /tmp/rahul/rahul-output

ied 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
13/03/14 18:12:29 INFO ipc.Client: Retrying connect to server: 10.94.214.188/10.94.214.188:54311. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
13/03/14 18:12:30 INFO ipc.Client: Retrying connect to server: 10.94.214.188/10.94.214.188:54311. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
13/03/14 18:12:31 INFO ipc.Client: Retrying connect to server: 10.94.214.188/10.94.214.188:54311. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
13/03/14 18:12:32 INFO ipc.Client: Retrying connect to server: 10.94.214.188/10.94.214.188:54311. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
Question:1. As I have been comparing HDFS and Lustre for Hadoop, what would be the right number of hardware nodes to compare?Say, I have 1 MDS, 2 OSS and 1 Lustre Client, on the other hand, 1 Namenode and 3 datanodes? How can I compare both FS?
Question:2. Do I really need 2 lustre client to setup Hadoop over Lustre? if it is possible, how can I use OSS and MDS too for Hadoop setup?
Question:3. As I read regarding the wordcount example, we need to insert data into HDFS filesystem, do we need to do same for Lustre too? Whats the command?
Question:4. What are the steps to confirm if HAdoop is actually using lustre FS?

Last edited by your_shadow03; 03-18-2013 at 12:49 AM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
PVFS2 vs Lustre on RocksCluster. abominable ROCK 0 06-11-2012 05:18 AM
Lustre server not healthy Langton Linux - General 1 05-13-2011 03:22 AM
Lustre server not healthy Langton Linux - Server 0 05-12-2011 04:15 AM
LXer: Lustre to run on ZFS LXer Syndicated Linux News 0 03-27-2008 12:11 PM
filsystem troubel... swoppe Red Hat 10 11-03-2003 03:39 PM


All times are GMT -5. The time now is 07:45 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration