about replication

Discussion:

about replication

Irfan Sayed

2013-08-05 06:53:20 UTC

hi,

i have setup the two node apache hadoop cluster on windows environment
one is namenode and another is datanode

everything is working fine.
one thing which i need to know is , how the replication starts
if i create a.txt in namenode , how it will be appeared in datanodes

please suggest

regards

Mohammad Tariq

2013-08-05 09:33:36 UTC

Permalink

Hello Irfan,

You can find all the answers from HDFS architecture
guide<http://hadoop.apache.org/docs/stable/hdfs_design.html>.
See the section Data
Organization<http://hadoop.apache.org/docs/stable/hdfs_design.html#Data+Organization>in
particular for this question.

Warm Regards,
Tariq
cloudfront.blogspot.com

Post by Irfan Sayed
hi,
i have setup the two node apache hadoop cluster on windows environment
one is namenode and another is datanode
everything is working fine.
one thing which i need to know is , how the replication starts
if i create a.txt in namenode , how it will be appeared in datanodes
please suggest
regards

Irfan Sayed

2013-08-05 11:16:05 UTC

Permalink

thanks mohammad
i ran the below command on NameNode

$ ./hadoop dfs -mkdir /wksp

and the "wksp" dir got created in c:\ ( as i have windows environment)

now , when i log in to one of the DataNode , then i am not able to see
c:\wksp

any issue ?
please suggest

regards

Post by Mohammad Tariq
Hello Irfan,
You can find all the answers from HDFS architecture guide<http://hadoop.apache.org/docs/stable/hdfs_design.html>.
See the section Data Organization<http://hadoop.apache.org/docs/stable/hdfs_design.html#Data+Organization>in particular for this question.
Warm Regards,
Tariq
cloudfront.blogspot.com

Mohammad Tariq

2013-08-05 11:48:07 UTC

Permalink

You cannot physically see the HDFS files and directories through local FS.
Either use HDFS shell or HDFS webUI(namenode_machine:50070).

Warm Regards,
Tariq
cloudfront.blogspot.com

Post by Irfan Sayed
thanks mohammad
i ran the below command on NameNode
$ ./hadoop dfs -mkdir /wksp
and the "wksp" dir got created in c:\ ( as i have windows environment)
now , when i log in to one of the DataNode , then i am not able to see
c:\wksp
any issue ?
please suggest
regards

Irfan Sayed

2013-08-05 11:59:12 UTC

Permalink

thanks.
please refer below:

***@DFS-DC /cygdrive/c/hadoop-1.1.2/hadoop-1.1.2/bin
$ ./hadoop dfs -ls /wksp
Found 1 items
drwxr-xr-x - Administrator Domain 0 2013-08-05 16:58 /wksp/New
folder

***@DFS-DC /cygdrive/c/hadoop-1.1.2/hadoop-1.1.2/bin
$

same command if i run on the datanode then it says:

***@DFS-1 /cygdrive/c/hadoop-1.1.2/hadoop-1.1.2/bin
$ ./hadoop dfs -ls /wksp
ls: Cannot access /wksp: No such file or directory.

does it mean that replication is not yet started ...?

please suggest

regards

Post by Mohammad Tariq
You cannot physically see the HDFS files and directories through local FS.
Either use HDFS shell or HDFS webUI(namenode_machine:50070).
Warm Regards,
Tariq
cloudfront.blogspot.com

manish dunani

2013-08-05 12:30:17 UTC

Permalink

You can not physically access the datanode.You have to understand it to
logically and it really happens.

Type "jps" command to check ur datanode was started or not.

when user stores the file into hdfs ,the request is goes to datanode and
datanode will divide the file into number of blocks.
Each block size is 64mb or 128mb.but,default is 64 mb.

The default replication factor is 1 that was u already set in ur
hdfs-site.xml .If u want to change ur replication factor from 1 to 3 for
particular file or particular directory in hdfs then use the below commands
appropriately.

First load any file to hdfs using command:

bin/hadoop dfs -copyFromLocal /ur/local/directory/path/to/file
/ur/hdfs/directory/path.

*commands:*

To set replication of an individual file to 4:

./bin/hadoop dfs -setrep -w 4 /path/to/file

You can also do this recursively. To change replication of entire HDFS to 1:

./bin/hadoop dfs -setrep -R -w 1 /
*
*
If the replication factor is 3 then data will divided into 3 blocks and
replicated to various data nodes of ur production cluster(multi node).

Here Replication means same file copied three times on data nodes for
handling hardware failure.

Post by Irfan Sayed
thanks.
$ ./hadoop dfs -ls /wksp
Found 1 items
drwxr-xr-x - Administrator Domain 0 2013-08-05 16:58 /wksp/New
folder
$
$ ./hadoop dfs -ls /wksp
ls: Cannot access /wksp: No such file or directory.
does it mean that replication is not yet started ...?
please suggest
regards

Post by Mohammad Tariq
You cannot physically see the HDFS files and directories through local
FS. Either use HDFS shell or HDFS webUI(namenode_machine:50070).
Warm Regards,
Tariq
cloudfront.blogspot.com

--
MANISH DUNANI
-THANX
+91 9426881954,+91 8460656443
manishd207-***@public.gmane.org

Irfan Sayed

2013-08-06 04:59:36 UTC

Permalink

thanks.
i verified, datanode is up and running

i ran the below command:

***@DFS-DC /cygdrive/c/hadoop-1.1.2/hadoop-1.1.2/bin
$ ./hadoop dfs -copyFromLocal
C:\\Users\\Administrator\\Desktop\\hadoop-1.1.2.tar /wksp
copyFromLocal: File C:/Users/Administrator/Desktop/hadoop-1.1.2.tar does
not exist.

it says , file does not exist. as my cluster is windows based , i dont know
how dir path needs to be used
i am using cygwin for linux style formatting

i tried below as well :

***@DFS-DC /cygdrive/c/hadoop-1.1.2/hadoop-1.1.2/bin
$ ./hadoop dfs -copyFromLocal
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar /wksp
copyFromLocal: File
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar does not exist.

***@DFS-DC /cygdrive/c/hadoop-1.1.2/hadoop-1.1.2/bin
$ ./hadoop dfs -copyFromLocal
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar.gz /wksp
copyFromLocal: File
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar.gz does not exist.

please suggest

regards

Post by manish dunani
You can not physically access the datanode.You have to understand it to
logically and it really happens.
Type "jps" command to check ur datanode was started or not.
when user stores the file into hdfs ,the request is goes to datanode and
datanode will divide the file into number of blocks.
Each block size is 64mb or 128mb.but,default is 64 mb.
The default replication factor is 1 that was u already set in ur
hdfs-site.xml .If u want to change ur replication factor from 1 to 3 for
particular file or particular directory in hdfs then use the below commands
appropriately.
bin/hadoop dfs -copyFromLocal /ur/local/directory/path/to/file
/ur/hdfs/directory/path.
*commands:*
./bin/hadoop dfs -setrep -w 4 /path/to/file
You can also do this recursively. To change replication of entire HDFS to
./bin/hadoop dfs -setrep -R -w 1 /
*
*
If the replication factor is 3 then data will divided into 3 blocks and
replicated to various data nodes of ur production cluster(multi node).
Here Replication means same file copied three times on data nodes for
handling hardware failure.

Post by Mohammad Tariq
You cannot physically see the HDFS files and directories through local
FS. Either use HDFS shell or HDFS webUI(namenode_machine:50070).
Warm Regards,
Tariq
cloudfront.blogspot.com

--
MANISH DUNANI
-THANX
+91 9426881954,+91 8460656443

manish dunani

2013-08-06 06:10:37 UTC

Permalink

hello,

I think u r newbie.You need to work on to learn from scratch.
Please do not mind.whatever u try to think hadoop should be like this, ur
concept is totally wrong.
but,your effort is positive.

My suggestion is try to learn on ubuntu os.Windows is not good enough for
hadoop.

Follow the link to setup single node cluster on ubuntu.

http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-single-node-cluster/

If u stop anywhere and do not get idea what to do next? just revert me .

Also read this:

https://docs.google.com/viewer?a=v&pid=sites&srcid=ZGVmYXVsdGRvbWFpbnxtYW5pc2hkdW5hbml8Z3g6NGRiM2JhODhhOGEzMTcyYw

*Thanks & Regards*
*
*
*Manish dunani*

Post by Irfan Sayed
thanks.
i verified, datanode is up and running
$ ./hadoop dfs -copyFromLocal
C:\\Users\\Administrator\\Desktop\\hadoop-1.1.2.tar /wksp
copyFromLocal: File C:/Users/Administrator/Desktop/hadoop-1.1.2.tar does
not exist.
it says , file does not exist. as my cluster is windows based , i dont
know how dir path needs to be used
i am using cygwin for linux style formatting
$ ./hadoop dfs -copyFromLocal
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar /wksp
copyFromLocal: File
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar does not exist.
$ ./hadoop dfs -copyFromLocal
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar.gz /wksp
copyFromLocal: File
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar.gz does not exist.
please suggest
regards

Post by Irfan Sayed
thanks.
$ ./hadoop dfs -ls /wksp
Found 1 items
drwxr-xr-x - Administrator Domain 0 2013-08-05 16:58
/wksp/New folder
$
$ ./hadoop dfs -ls /wksp
ls: Cannot access /wksp: No such file or directory.
does it mean that replication is not yet started ...?
please suggest
regards

Post by Mohammad Tariq
You cannot physically see the HDFS files and directories through local
FS. Either use HDFS shell or HDFS webUI(namenode_machine:50070).
Warm Regards,
Tariq
cloudfront.blogspot.com

--
MANISH DUNANI
-THANX
+91 9426881954,+91 8460656443

--
MANISH DUNANI
-THANX
+91 9426881954,+91 8460656443
manishd207-***@public.gmane.org

Irfan Sayed

2013-08-06 06:19:39 UTC

Permalink

thanks. yes , i am newbie.
however, i need windows setup.

let me surely refer the doc and link which u sent but i need this to be
working ...
can you please help

regards

Irfan Sayed

2013-08-06 06:50:31 UTC

Permalink

hi,
i have gone high level through the doc and seems very promising. really
nice
but can you please help me on this issue .

i need to do PoC first and get the demo out

regards

Post by Irfan Sayed
thanks. yes , i am newbie.
however, i need windows setup.
let me surely refer the doc and link which u sent but i need this to be
working ...
can you please help
regards

manish dunani

2013-08-06 07:10:47 UTC

Permalink

*You are wrong at this:*

***@DFS-DC /cygdrive/c/hadoop-1.1.2/hadoop-1.1.2/bin
$ ./hadoop dfs -copyFromLocal
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar /wksp
copyFromLocal: File
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar does not exist.

***@DFS-DC /cygdrive/c/hadoop-1.1.2/hadoop-1.1.2/bin
$ ./hadoop dfs -copyFromLocal
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar.gz /wksp
copyFromLocal: File
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar.gz does not exist.

Because,You had wrote both the paths local and You need not to copy hadoop
into hdfs...Hadoop is already working..

Just check out in browser by after starting ur single node cluster :

localhost:50070

then go for browse the filesystem link in it..

If there is no directory then make directory there.
That is your hdfs directory.
Then copy any text file there(no need to copy hadoop there).beacause u are
going to do processing on that data in text file.That's why hadoop is used
for ,first u need to make it clear in ur mind.Then and then u will do
it...otherwise not possible..

*Try this: *

***@DFS-DC /cygdrive/c/hadoop-1.1.2/hadoop-1.1.2
$ .bin/hadoop dfs -copyFromLocal /full/local/path/to/ur/file
/hdfs/directory/path

Post by Irfan Sayed
thanks. yes , i am newbie.
however, i need windows setup.
let me surely refer the doc and link which u sent but i need this to be
working ...
can you please help
regards

--
MANISH DUNANI
-THANX
+91 9426881954,+91 8460656443
manishd207-***@public.gmane.org

Irfan Sayed

2013-08-06 11:39:43 UTC

Permalink

thanks.
when i browse the file system , i am getting following :
i haven't seen any make directory option there

i need to create it from command line ?
further, in the hdfs-site.xml file , i have given following entries. are
they correct ?

<property>
<name>dfs.data.dir</name>
<value>c:\\wksp</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>c:\\wksp</value>
</property>

please suggest

[image: Inline image 1]

Post by manish dunani
*You are wrong at this:*
$ ./hadoop dfs -copyFromLocal
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar /wksp
copyFromLocal: File
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar does not exist.
$ ./hadoop dfs -copyFromLocal
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar.gz /wksp
copyFromLocal: File
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar.gz does not exist.
Because,You had wrote both the paths local and You need not to copy hadoop
into hdfs...Hadoop is already working..
localhost:50070
then go for browse the filesystem link in it..
If there is no directory then make directory there.
That is your hdfs directory.
Then copy any text file there(no need to copy hadoop there).beacause u are
going to do processing on that data in text file.That's why hadoop is used
for ,first u need to make it clear in ur mind.Then and then u will do
it...otherwise not possible..
*Try this: *
$ .bin/hadoop dfs -copyFromLocal /full/local/path/to/ur/file
/hdfs/directory/path

Post by Irfan Sayed
thanks. yes , i am newbie.
however, i need windows setup.
let me surely refer the doc and link which u sent but i need this to be
working ...
can you please help
regards

--
MANISH DUNANI
-THANX
+91 9426881954,+91 8460656443

Mohammad Tariq

2013-08-06 11:49:13 UTC

Permalink

Hello Irfan,

Sorry for being unresponsive. Got stuck with some imp work.

HDFS webUI doesn't provide us the ability to create file or directory. You
can browse HDFS, view files, download files etc. But operation like create,
move, copy etc are not supported.

These values look fine to me.

One suggestion though. Try getting a Linux machine(if possible). Or at
least use a VM. I personally feel that using Hadoop on windows is always
messy.

Warm Regards,
Tariq
cloudfront.blogspot.com

Post by Irfan Sayed
thanks.
i haven't seen any make directory option there
i need to create it from command line ?
further, in the hdfs-site.xml file , i have given following entries. are
they correct ?
<property>
<name>dfs.data.dir</name>
<value>c:\\wksp</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>c:\\wksp</value>
</property>
please suggest
[image: Inline image 1]

Post by manish dunani
*You are wrong at this:*
$ ./hadoop dfs -copyFromLocal
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar /wksp
copyFromLocal: File
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar does not exist.
$ ./hadoop dfs -copyFromLocal
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar.gz /wksp
copyFromLocal: File
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar.gz does not exist.
Because,You had wrote both the paths local and You need not to copy
hadoop into hdfs...Hadoop is already working..
localhost:50070
then go for browse the filesystem link in it..
If there is no directory then make directory there.
That is your hdfs directory.
Then copy any text file there(no need to copy hadoop there).beacause u
are going to do processing on that data in text file.That's why hadoop is
used for ,first u need to make it clear in ur mind.Then and then u will do
it...otherwise not possible..
*Try this: *
$ .bin/hadoop dfs -copyFromLocal /full/local/path/to/ur/file
/hdfs/directory/path

Post by Irfan Sayed
thanks. yes , i am newbie.
however, i need windows setup.
let me surely refer the doc and link which u sent but i need this to be
working ...
can you please help
regards

--
MANISH DUNANI
-THANX
+91 9426881954,+91 8460656443

Irfan Sayed

2013-08-06 11:56:09 UTC

Permalink

thanks.
however, i need this to be working on windows environment as project
requirement.
i will add/work on Linux later

so, now , at this stage , c:\\wksp is the HDFS file system OR do i need to
create it from command line ?

please suggest

regards,

Post by Mohammad Tariq
Hello Irfan,
Sorry for being unresponsive. Got stuck with some imp work.
HDFS webUI doesn't provide us the ability to create file or directory. You
can browse HDFS, view files, download files etc. But operation like create,
move, copy etc are not supported.
These values look fine to me.
One suggestion though. Try getting a Linux machine(if possible). Or at
least use a VM. I personally feel that using Hadoop on windows is always
messy.
Warm Regards,
Tariq
cloudfront.blogspot.com

Post by manish dunani
*You are wrong at this:*
$ ./hadoop dfs -copyFromLocal
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar /wksp
copyFromLocal: File
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar does not exist.
$ ./hadoop dfs -copyFromLocal
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar.gz /wksp
copyFromLocal: File
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar.gz does not exist.
Because,You had wrote both the paths local and You need not to copy
hadoop into hdfs...Hadoop is already working..
localhost:50070
then go for browse the filesystem link in it..
If there is no directory then make directory there.
That is your hdfs directory.
Then copy any text file there(no need to copy hadoop there).beacause u
are going to do processing on that data in text file.That's why hadoop is
used for ,first u need to make it clear in ur mind.Then and then u will do
it...otherwise not possible..
*Try this: *
$ .bin/hadoop dfs -copyFromLocal /full/local/path/to/ur/file
/hdfs/directory/path

Post by Irfan Sayed
thanks. yes , i am newbie.
however, i need windows setup.
let me surely refer the doc and link which u sent but i need this to be
working ...
can you please help
regards

--
MANISH DUNANI
-THANX
+91 9426881954,+91 8460656443

Mohammad Tariq

2013-08-06 12:00:51 UTC

Permalink

Create 2 directories manually corresponding to the values of dfs.name.dir
and dfs.data.dir properties and change the permissions of these directories
to 755. When you start pushing data into your HDFS, data will start going
inside the directory specified by dfs.data.dir and the associated metadata
will go inside dfs.name.dir. Remember, you store data in HDFS, but it
eventually gets stored in your local/native FS. But you cannot see this
data directly on your local/native FS.

Warm Regards,
Tariq
cloudfront.blogspot.com

Post by Irfan Sayed
thanks.
however, i need this to be working on windows environment as project
requirement.
i will add/work on Linux later
so, now , at this stage , c:\\wksp is the HDFS file system OR do i need to
create it from command line ?
please suggest
regards,

Post by Mohammad Tariq
Hello Irfan,
Sorry for being unresponsive. Got stuck with some imp work.
HDFS webUI doesn't provide us the ability to create file or directory.
You can browse HDFS, view files, download files etc. But operation like
create, move, copy etc are not supported.
These values look fine to me.
One suggestion though. Try getting a Linux machine(if possible). Or at
least use a VM. I personally feel that using Hadoop on windows is always
messy.
Warm Regards,
Tariq
cloudfront.blogspot.com

Post by manish dunani
*You are wrong at this:*
$ ./hadoop dfs -copyFromLocal
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar /wksp
copyFromLocal: File
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar does not exist.
$ ./hadoop dfs -copyFromLocal
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar.gz /wksp
copyFromLocal: File
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar.gz does not exist.
Because,You had wrote both the paths local and You need not to copy
hadoop into hdfs...Hadoop is already working..
localhost:50070
then go for browse the filesystem link in it..
If there is no directory then make directory there.
That is your hdfs directory.
Then copy any text file there(no need to copy hadoop there).beacause u
are going to do processing on that data in text file.That's why hadoop is
used for ,first u need to make it clear in ur mind.Then and then u will do
it...otherwise not possible..
*Try this: *
$ .bin/hadoop dfs -copyFromLocal /full/local/path/to/ur/file
/hdfs/directory/path

Post by Irfan Sayed
thanks. yes , i am newbie.
however, i need windows setup.
let me surely refer the doc and link which u sent but i need this to
be working ...
can you please help
regards

--
MANISH DUNANI
-THANX
+91 9426881954,+91 8460656443

Irfan Sayed

2013-08-06 17:05:17 UTC

Permalink

these dir needs to be created on all datanodes and namenodes ?
further, hdfs-site.xml needs to be updated on both datanodes and namenodes
for these new dir?

regards

Post by Mohammad Tariq
Create 2 directories manually corresponding to the values of dfs.name.dir
and dfs.data.dir properties and change the permissions of these directories
to 755. When you start pushing data into your HDFS, data will start going
inside the directory specified by dfs.data.dir and the associated metadata
will go inside dfs.name.dir. Remember, you store data in HDFS, but it
eventually gets stored in your local/native FS. But you cannot see this
data directly on your local/native FS.
Warm Regards,
Tariq
cloudfront.blogspot.com

Post by Irfan Sayed
thanks.
however, i need this to be working on windows environment as project
requirement.
i will add/work on Linux later
so, now , at this stage , c:\\wksp is the HDFS file system OR do i need
to create it from command line ?
please suggest
regards,

Post by Mohammad Tariq
Hello Irfan,
Sorry for being unresponsive. Got stuck with some imp work.
HDFS webUI doesn't provide us the ability to create file or directory.
You can browse HDFS, view files, download files etc. But operation like
create, move, copy etc are not supported.
These values look fine to me.
One suggestion though. Try getting a Linux machine(if possible). Or at
least use a VM. I personally feel that using Hadoop on windows is always
messy.
Warm Regards,
Tariq
cloudfront.blogspot.com

Post by Irfan Sayed
thanks.
i haven't seen any make directory option there
i need to create it from command line ?
further, in the hdfs-site.xml file , i have given following entries.
are they correct ?
<property>
<name>dfs.data.dir</name>
<value>c:\\wksp</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>c:\\wksp</value>
</property>
please suggest
[image: Inline image 1]

Post by manish dunani
*You are wrong at this:*
$ ./hadoop dfs -copyFromLocal
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar /wksp
copyFromLocal: File
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar does not exist.
$ ./hadoop dfs -copyFromLocal
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar.gz /wksp
copyFromLocal: File
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar.gz does not exist.
Because,You had wrote both the paths local and You need not to copy
hadoop into hdfs...Hadoop is already working..
localhost:50070
then go for browse the filesystem link in it..
If there is no directory then make directory there.
That is your hdfs directory.
Then copy any text file there(no need to copy hadoop there).beacause u
are going to do processing on that data in text file.That's why hadoop is
used for ,first u need to make it clear in ur mind.Then and then u will do
it...otherwise not possible..
*Try this: *
$ .bin/hadoop dfs -copyFromLocal /full/local/path/to/ur/file
/hdfs/directory/path

Post by Irfan Sayed
thanks. yes , i am newbie.
however, i need windows setup.
let me surely refer the doc and link which u sent but i need this to
be working ...
can you please help
regards

--
MANISH DUNANI
-THANX
+91 9426881954,+91 8460656443

Irfan Sayed

2013-08-06 17:29:28 UTC

Permalink

i have created these dir "wksp_data" and "wksp_name" on both datanode and
namenode
made the respective changes in "hdfs-site.xml" file
formatted the namenode
started the dfs

but still, not able to browse the file system through web browser
please refer below

anything still missing ?
please suggest

[image: Inline image 1]

Post by Irfan Sayed
these dir needs to be created on all datanodes and namenodes ?
further, hdfs-site.xml needs to be updated on both datanodes and
namenodes for these new dir?
regards

Post by Irfan Sayed
thanks.
however, i need this to be working on windows environment as project
requirement.
i will add/work on Linux later
so, now , at this stage , c:\\wksp is the HDFS file system OR do i need
to create it from command line ?
please suggest
regards,

Post by Mohammad Tariq
Hello Irfan,
Sorry for being unresponsive. Got stuck with some imp work.
HDFS webUI doesn't provide us the ability to create file or directory.
You can browse HDFS, view files, download files etc. But operation like
create, move, copy etc are not supported.
These values look fine to me.
One suggestion though. Try getting a Linux machine(if possible). Or at
least use a VM. I personally feel that using Hadoop on windows is always
messy.
Warm Regards,
Tariq
cloudfront.blogspot.com

Post by Irfan Sayed
thanks.
i haven't seen any make directory option there
i need to create it from command line ?
further, in the hdfs-site.xml file , i have given following entries.
are they correct ?
<property>
<name>dfs.data.dir</name>
<value>c:\\wksp</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>c:\\wksp</value>
</property>
please suggest
[image: Inline image 1]

Post by manish dunani
*You are wrong at this:*
$ ./hadoop dfs -copyFromLocal
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar /wksp
copyFromLocal: File
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar does not exist.
$ ./hadoop dfs -copyFromLocal
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar.gz /wksp
copyFromLocal: File
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar.gz does not exist.
Because,You had wrote both the paths local and You need not to copy
hadoop into hdfs...Hadoop is already working..
localhost:50070
then go for browse the filesystem link in it..
If there is no directory then make directory there.
That is your hdfs directory.
Then copy any text file there(no need to copy hadoop there).beacause
u are going to do processing on that data in text file.That's why hadoop is
used for ,first u need to make it clear in ur mind.Then and then u will do
it...otherwise not possible..
*Try this: *
$ .bin/hadoop dfs -copyFromLocal /full/local/path/to/ur/file
/hdfs/directory/path

Post by Irfan Sayed
thanks. yes , i am newbie.
however, i need windows setup.
let me surely refer the doc and link which u sent but i need this to
be working ...
can you please help
regards

--
MANISH DUNANI
-THANX
+91 9426881954,+91 8460656443

Mohammad Tariq

2013-08-06 20:22:24 UTC

Permalink

OK. we'll start fresh. Could you plz show me your latest config files?

BTW, are your daemons running fine?Use JPS to verify that.

Warm Regards,
Tariq
cloudfront.blogspot.com

Post by Irfan Sayed
i have created these dir "wksp_data" and "wksp_name" on both datanode and
namenode
made the respective changes in "hdfs-site.xml" file
formatted the namenode
started the dfs
but still, not able to browse the file system through web browser
please refer below
anything still missing ?
please suggest
[image: Inline image 1]

Post by Irfan Sayed
these dir needs to be created on all datanodes and namenodes ?
further, hdfs-site.xml needs to be updated on both datanodes and
namenodes for these new dir?
regards

Post by Mohammad Tariq
Create 2 directories manually corresponding to the values of
dfs.name.dir and dfs.data.dir properties and change the permissions of
these directories to 755. When you start pushing data into your HDFS, data
will start going inside the directory specified by dfs.data.dir and the
associated metadata will go inside dfs.name.dir. Remember, you store data
in HDFS, but it eventually gets stored in your local/native FS. But you
cannot see this data directly on your local/native FS.
Warm Regards,
Tariq
cloudfront.blogspot.com

Post by Irfan Sayed
thanks.
however, i need this to be working on windows environment as project
requirement.
i will add/work on Linux later
so, now , at this stage , c:\\wksp is the HDFS file system OR do i need
to create it from command line ?
please suggest
regards,

Post by Mohammad Tariq
Hello Irfan,
Sorry for being unresponsive. Got stuck with some imp work.
HDFS webUI doesn't provide us the ability to create file or directory.
You can browse HDFS, view files, download files etc. But operation like
create, move, copy etc are not supported.
These values look fine to me.
One suggestion though. Try getting a Linux machine(if possible). Or at
least use a VM. I personally feel that using Hadoop on windows is always
messy.
Warm Regards,
Tariq
cloudfront.blogspot.com

Post by Irfan Sayed
thanks.
i haven't seen any make directory option there
i need to create it from command line ?
further, in the hdfs-site.xml file , i have given following entries.
are they correct ?
<property>
<name>dfs.data.dir</name>
<value>c:\\wksp</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>c:\\wksp</value>
</property>
please suggest
[image: Inline image 1]

Post by manish dunani
*You are wrong at this:*
$ ./hadoop dfs -copyFromLocal
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar /wksp
copyFromLocal: File
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar does not exist.
$ ./hadoop dfs -copyFromLocal
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar.gz /wksp
copyFromLocal: File
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar.gz does not exist.
Because,You had wrote both the paths local and You need not to copy
hadoop into hdfs...Hadoop is already working..
localhost:50070
then go for browse the filesystem link in it..
If there is no directory then make directory there.
That is your hdfs directory.
Then copy any text file there(no need to copy hadoop there).beacause
u are going to do processing on that data in text file.That's why hadoop is
used for ,first u need to make it clear in ur mind.Then and then u will do
it...otherwise not possible..
*Try this: *
$ .bin/hadoop dfs -copyFromLocal /full/local/path/to/ur/file
/hdfs/directory/path

Post by Irfan Sayed
thanks. yes , i am newbie.
however, i need windows setup.
let me surely refer the doc and link which u sent but i need this
to be working ...
can you please help
regards

--
MANISH DUNANI
-THANX
+91 9426881954,+91 8460656443

Irfan Sayed

2013-08-07 04:11:55 UTC

Permalink

thanks.

if i run the jps command on namenode :

***@DFS-DC /cygdrive/c/Java/jdk1.7.0_25/bin
$ ./jps.exe
3164 NameNode
1892 Jps

same command on datanode :

***@DFS-1 /cygdrive/c/Java/jdk1.7.0_25/bin
$ ./jps.exe
3848 Jps

jps does not list any process for datanode. however, on web browser i can
see one live data node
please find the attached conf rar file of namenode

regards

Post by Mohammad Tariq
OK. we'll start fresh. Could you plz show me your latest config files?
BTW, are your daemons running fine?Use JPS to verify that.
Warm Regards,
Tariq
cloudfront.blogspot.com

Post by Irfan Sayed
these dir needs to be created on all datanodes and namenodes ?
further, hdfs-site.xml needs to be updated on both datanodes and
namenodes for these new dir?
regards

Post by Mohammad Tariq
Create 2 directories manually corresponding to the values of
dfs.name.dir and dfs.data.dir properties and change the permissions of
these directories to 755. When you start pushing data into your HDFS, data
will start going inside the directory specified by dfs.data.dir and the
associated metadata will go inside dfs.name.dir. Remember, you store data
in HDFS, but it eventually gets stored in your local/native FS. But you
cannot see this data directly on your local/native FS.
Warm Regards,
Tariq
cloudfront.blogspot.com

Post by Irfan Sayed
thanks.
however, i need this to be working on windows environment as project
requirement.
i will add/work on Linux later
so, now , at this stage , c:\\wksp is the HDFS file system OR do i
need to create it from command line ?
please suggest
regards,

Post by Mohammad Tariq
Hello Irfan,
Sorry for being unresponsive. Got stuck with some imp work.
HDFS webUI doesn't provide us the ability to create file or
directory. You can browse HDFS, view files, download files etc. But
operation like create, move, copy etc are not supported.
These values look fine to me.
One suggestion though. Try getting a Linux machine(if possible). Or
at least use a VM. I personally feel that using Hadoop on windows is always
messy.
Warm Regards,
Tariq
cloudfront.blogspot.com

Post by Irfan Sayed
thanks.
i haven't seen any make directory option there
i need to create it from command line ?
further, in the hdfs-site.xml file , i have given following entries.
are they correct ?
<property>
<name>dfs.data.dir</name>
<value>c:\\wksp</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>c:\\wksp</value>
</property>
please suggest
[image: Inline image 1]

Post by manish dunani
*You are wrong at this:*
$ ./hadoop dfs -copyFromLocal
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar /wksp
copyFromLocal: File
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar does not exist.
$ ./hadoop dfs -copyFromLocal
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar.gz /wksp
copyFromLocal: File
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar.gz does not exist.
Because,You had wrote both the paths local and You need not to copy
hadoop into hdfs...Hadoop is already working..
localhost:50070
then go for browse the filesystem link in it..
If there is no directory then make directory there.
That is your hdfs directory.
Then copy any text file there(no need to copy hadoop
there).beacause u are going to do processing on that data in text
file.That's why hadoop is used for ,first u need to make it clear in ur
mind.Then and then u will do it...otherwise not possible..
*Try this: *
$ .bin/hadoop dfs -copyFromLocal /full/local/path/to/ur/file
/hdfs/directory/path

Post by Irfan Sayed
thanks. yes , i am newbie.
however, i need windows setup.
let me surely refer the doc and link which u sent but i need this
to be working ...
can you please help
regards

--
MANISH DUNANI
-THANX
+91 9426881954,+91 8460656443

Irfan Sayed

2013-08-07 04:19:23 UTC

Permalink

attachment got quarantined
resending in txt format. please rename it to conf.rar

regards

Post by Irfan Sayed
thanks.
$ ./jps.exe
3164 NameNode
1892 Jps
$ ./jps.exe
3848 Jps
jps does not list any process for datanode. however, on web browser i can
see one live data node
please find the attached conf rar file of namenode
regards

Post by Mohammad Tariq
OK. we'll start fresh. Could you plz show me your latest config files?
BTW, are your daemons running fine?Use JPS to verify that.
Warm Regards,
Tariq
cloudfront.blogspot.com

Post by Irfan Sayed
i have created these dir "wksp_data" and "wksp_name" on both datanode
and namenode
made the respective changes in "hdfs-site.xml" file
formatted the namenode
started the dfs
but still, not able to browse the file system through web browser
please refer below
anything still missing ?
please suggest
[image: Inline image 1]

Post by Irfan Sayed
these dir needs to be created on all datanodes and namenodes ?
further, hdfs-site.xml needs to be updated on both datanodes and
namenodes for these new dir?
regards

Post by Mohammad Tariq
Create 2 directories manually corresponding to the values of
dfs.name.dir and dfs.data.dir properties and change the permissions of
these directories to 755. When you start pushing data into your HDFS, data
will start going inside the directory specified by dfs.data.dir and the
associated metadata will go inside dfs.name.dir. Remember, you store data
in HDFS, but it eventually gets stored in your local/native FS. But you
cannot see this data directly on your local/native FS.
Warm Regards,
Tariq
cloudfront.blogspot.com

Post by Irfan Sayed
thanks.
however, i need this to be working on windows environment as project
requirement.
i will add/work on Linux later
so, now , at this stage , c:\\wksp is the HDFS file system OR do i
need to create it from command line ?
please suggest
regards,

Post by Mohammad Tariq
Hello Irfan,
Sorry for being unresponsive. Got stuck with some imp work.
HDFS webUI doesn't provide us the ability to create file or
directory. You can browse HDFS, view files, download files etc. But
operation like create, move, copy etc are not supported.
These values look fine to me.
One suggestion though. Try getting a Linux machine(if possible). Or
at least use a VM. I personally feel that using Hadoop on windows is always
messy.
Warm Regards,
Tariq
cloudfront.blogspot.com

Post by Irfan Sayed
thanks.
i haven't seen any make directory option there
i need to create it from command line ?
further, in the hdfs-site.xml file , i have given following
entries. are they correct ?
<property>
<name>dfs.data.dir</name>
<value>c:\\wksp</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>c:\\wksp</value>
</property>
please suggest
[image: Inline image 1]
On Tue, Aug 6, 2013 at 12:40 PM, manish dunani <

Post by manish dunani
*You are wrong at this:*
$ ./hadoop dfs -copyFromLocal
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar /wksp
copyFromLocal: File
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar does not exist.
$ ./hadoop dfs -copyFromLocal
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar.gz /wksp
copyFromLocal: File
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar.gz does not exist.
Because,You had wrote both the paths local and You need not to
copy hadoop into hdfs...Hadoop is already working..
Just check out in browser by after starting ur single node cluster
localhost:50070
then go for browse the filesystem link in it..
If there is no directory then make directory there.
That is your hdfs directory.
Then copy any text file there(no need to copy hadoop
there).beacause u are going to do processing on that data in text
file.That's why hadoop is used for ,first u need to make it clear in ur
mind.Then and then u will do it...otherwise not possible..
*Try this: *
$ .bin/hadoop dfs -copyFromLocal /full/local/path/to/ur/file
/hdfs/directory/path

Post by Irfan Sayed
thanks. yes , i am newbie.
however, i need windows setup.
let me surely refer the doc and link which u sent but i need this
to be working ...
can you please help
regards

--
MANISH DUNANI
-THANX
+91 9426881954,+91 8460656443

Irfan Sayed

2013-08-08 00:26:26 UTC

Permalink

please suggest

regards

Post by Irfan Sayed
attachment got quarantined
resending in txt format. please rename it to conf.rar
regards

Post by Mohammad Tariq
OK. we'll start fresh. Could you plz show me your latest config files?
BTW, are your daemons running fine?Use JPS to verify that.
Warm Regards,
Tariq
cloudfront.blogspot.com

Post by Irfan Sayed
i have created these dir "wksp_data" and "wksp_name" on both datanode
and namenode
made the respective changes in "hdfs-site.xml" file
formatted the namenode
started the dfs
but still, not able to browse the file system through web browser
please refer below
anything still missing ?
please suggest
[image: Inline image 1]

Post by Irfan Sayed
these dir needs to be created on all datanodes and namenodes ?
further, hdfs-site.xml needs to be updated on both datanodes and
namenodes for these new dir?
regards

Post by Mohammad Tariq
Create 2 directories manually corresponding to the values of
dfs.name.dir and dfs.data.dir properties and change the permissions of
these directories to 755. When you start pushing data into your HDFS, data
will start going inside the directory specified by dfs.data.dir and the
associated metadata will go inside dfs.name.dir. Remember, you store data
in HDFS, but it eventually gets stored in your local/native FS. But you
cannot see this data directly on your local/native FS.
Warm Regards,
Tariq
cloudfront.blogspot.com

Post by Irfan Sayed
thanks.
however, i need this to be working on windows environment as project
requirement.
i will add/work on Linux later
so, now , at this stage , c:\\wksp is the HDFS file system OR do i
need to create it from command line ?
please suggest
regards,

Post by Mohammad Tariq
Hello Irfan,
Sorry for being unresponsive. Got stuck with some imp work.
HDFS webUI doesn't provide us the ability to create file or
directory. You can browse HDFS, view files, download files etc. But
operation like create, move, copy etc are not supported.
These values look fine to me.
One suggestion though. Try getting a Linux machine(if possible). Or
at least use a VM. I personally feel that using Hadoop on windows is always
messy.
Warm Regards,
Tariq
cloudfront.blogspot.com

Post by Irfan Sayed
thanks.
i haven't seen any make directory option there
i need to create it from command line ?
further, in the hdfs-site.xml file , i have given following
entries. are they correct ?
<property>
<name>dfs.data.dir</name>
<value>c:\\wksp</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>c:\\wksp</value>
</property>
please suggest
[image: Inline image 1]
On Tue, Aug 6, 2013 at 12:40 PM, manish dunani <

Post by manish dunani
*You are wrong at this:*
$ ./hadoop dfs -copyFromLocal
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar /wksp
copyFromLocal: File
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar does not exist.
$ ./hadoop dfs -copyFromLocal
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar.gz /wksp
copyFromLocal: File
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar.gz does not exist.
Because,You had wrote both the paths local and You need not to
copy hadoop into hdfs...Hadoop is already working..
Just check out in browser by after starting ur single node
localhost:50070
then go for browse the filesystem link in it..
If there is no directory then make directory there.
That is your hdfs directory.
Then copy any text file there(no need to copy hadoop
there).beacause u are going to do processing on that data in text
file.That's why hadoop is used for ,first u need to make it clear in ur
mind.Then and then u will do it...otherwise not possible..
*Try this: *
$ .bin/hadoop dfs -copyFromLocal /full/local/path/to/ur/file
/hdfs/directory/path
On Tue, Aug 6, 2013 at 11:49 AM, Irfan Sayed <

Post by Irfan Sayed
thanks. yes , i am newbie.
however, i need windows setup.
let me surely refer the doc and link which u sent but i need
this to be working ...
can you please help
regards

--
MANISH DUNANI
-THANX
+91 9426881954,+91 8460656443

Irfan Sayed

2013-08-13 07:26:22 UTC

Permalink

hey Tariq,
i am still stuck ..
can you please suggest

regards
irfan

Post by Irfan Sayed
please suggest
regards

Post by Irfan Sayed
attachment got quarantined
resending in txt format. please rename it to conf.rar
regards

Post by Irfan Sayed
thanks.
$ ./jps.exe
3164 NameNode
1892 Jps
$ ./jps.exe
3848 Jps
jps does not list any process for datanode. however, on web browser i
can see one live data node
please find the attached conf rar file of namenode
regards

Post by Mohammad Tariq
OK. we'll start fresh. Could you plz show me your latest config files?
BTW, are your daemons running fine?Use JPS to verify that.
Warm Regards,
Tariq
cloudfront.blogspot.com

Post by Irfan Sayed
i have created these dir "wksp_data" and "wksp_name" on both datanode
and namenode
made the respective changes in "hdfs-site.xml" file
formatted the namenode
started the dfs
but still, not able to browse the file system through web browser
please refer below
anything still missing ?
please suggest
[image: Inline image 1]

Post by Irfan Sayed
these dir needs to be created on all datanodes and namenodes ?
further, hdfs-site.xml needs to be updated on both datanodes and
namenodes for these new dir?
regards

Post by Mohammad Tariq
Create 2 directories manually corresponding to the values of
dfs.name.dir and dfs.data.dir properties and change the permissions of
these directories to 755. When you start pushing data into your HDFS, data
will start going inside the directory specified by dfs.data.dir and the
associated metadata will go inside dfs.name.dir. Remember, you store data
in HDFS, but it eventually gets stored in your local/native FS. But you
cannot see this data directly on your local/native FS.
Warm Regards,
Tariq
cloudfront.blogspot.com

Post by Irfan Sayed
thanks.
however, i need this to be working on windows environment as
project requirement.
i will add/work on Linux later
so, now , at this stage , c:\\wksp is the HDFS file system OR do i
need to create it from command line ?
please suggest
regards,

Post by Mohammad Tariq
Hello Irfan,
Sorry for being unresponsive. Got stuck with some imp work.
HDFS webUI doesn't provide us the ability to create file or
directory. You can browse HDFS, view files, download files etc. But
operation like create, move, copy etc are not supported.
These values look fine to me.
One suggestion though. Try getting a Linux machine(if possible).
Or at least use a VM. I personally feel that using Hadoop on windows is
always messy.
Warm Regards,
Tariq
cloudfront.blogspot.com

Post by Irfan Sayed
thanks.
i haven't seen any make directory option there
i need to create it from command line ?
further, in the hdfs-site.xml file , i have given following
entries. are they correct ?
<property>
<name>dfs.data.dir</name>
<value>c:\\wksp</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>c:\\wksp</value>
</property>
please suggest
[image: Inline image 1]
On Tue, Aug 6, 2013 at 12:40 PM, manish dunani <

Post by manish dunani
*You are wrong at this:*
$ ./hadoop dfs -copyFromLocal
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar /wksp
copyFromLocal: File
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar does not exist.
$ ./hadoop dfs -copyFromLocal
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar.gz /wksp
copyFromLocal: File
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar.gz does not exist.
Because,You had wrote both the paths local and You need not to
copy hadoop into hdfs...Hadoop is already working..
Just check out in browser by after starting ur single node
localhost:50070
then go for browse the filesystem link in it..
If there is no directory then make directory there.
That is your hdfs directory.
Then copy any text file there(no need to copy hadoop
there).beacause u are going to do processing on that data in text
file.That's why hadoop is used for ,first u need to make it clear in ur
mind.Then and then u will do it...otherwise not possible..
*Try this: *
$ .bin/hadoop dfs -copyFromLocal /full/local/path/to/ur/file
/hdfs/directory/path
On Tue, Aug 6, 2013 at 11:49 AM, Irfan Sayed <

Post by Irfan Sayed
thanks. yes , i am newbie.
however, i need windows setup.
let me surely refer the doc and link which u sent but i need
this to be working ...
can you please help
regards

--
MANISH DUNANI
-THANX
+91 9426881954,+91 8460656443

Irfan Sayed

2013-08-19 04:15:03 UTC

Permalink

please suggest

regards
irfan

Post by Irfan Sayed
hey Tariq,
i am still stuck ..
can you please suggest
regards
irfan