Here's your issue (from the logs you attached earlier):
CAQuietExec: Checking JAVA_HOME is set correctly...
CAQuietExec: Files\Java\jdk1.6.0_31 was unexpected at this time.
It seems that you installed Java prerequisite in the default path, which is %PROGRAMFILES% (expands to C:\Program Files in your case). HDP 1.3 does not like spaces in paths, do you need to reinstall Java under c:\java\ or something similar (in a path with no spaces).
From: Irfan Sayed [mailto:irfu.sayed-***@public.gmane.org]
Sent: Thursday, September 5, 2013 8:42 PM
To: user-7ArZoLwFLBtd/SJB6HiN2Ni2O/***@public.gmane.org
Subject: Re: about replication
please find the attached.
i don't have "c:\HadoopInstallFiles\HadoopSetupTools\hdp-1.3.0.0.winpkg.install.log" as it is not generated
regards
irfan
On Thu, Sep 5, 2013 at 6:09 PM, Olivier Renault <orenault-RYNwJFaOa9CEK/***@public.gmane.org<mailto:orenault-RYNwJFaOa9CEK/***@public.gmane.org>> wrote:
Could you share the log files ( c:\hdp.log, c:\HadoopInstallFiles\HadoopSetupTools\hdp-1.3.0.0.winpkg.install.log ) as well as your clusterproperties.txt ?
Thanks,
Olivier
On 5 September 2013 12:33, Irfan Sayed <irfu.sayed-***@public.gmane.org<mailto:irfu.sayed-***@public.gmane.org>> wrote:
thanks. i followed the user manual for deployment and installed all pre-requisites
i modified the command and still the issue persist. please suggest
please refer below
[Inline image 1]
regards
irfan
On Wed, Sep 4, 2013 at 5:13 PM, Olivier Renault <orenault-RYNwJFaOa9CEK/***@public.gmane.org<mailto:orenault-RYNwJFaOa9CEK/***@public.gmane.org>> wrote:
The command to install it is msiexec /i msifile /...
You will find the correct syntax as part of doc.
Happy reading
Olivier
On 4 Sep 2013 12:37, "Irfan Sayed" <irfu.sayed-***@public.gmane.org<mailto:***@gmail.com>> wrote:
thanks.
i referred the logs and manuals. i modified the clusterproperties file and then double click on the msi file
however, it still failed.
further i started the installation on command line by giving HDP_LAYOUT=clusterproperties file path,
installation went ahead and it failed for .NET framework 4.0 and VC++ redistributable package dependency
i installed both and started again the installation.
failed again with following error
[Inline image 1]
when i search for the logs mentioned in the error , i never found that
please suggest
regards
irfan
On Tue, Sep 3, 2013 at 12:58 PM, Olivier Renault <orenault-RYNwJFaOa9CEK/***@public.gmane.org<mailto:orenault-RYNwJFaOa9CEK/***@public.gmane.org>> wrote:
Correct, you need to define the cluster configuration as part of a file. You will find some information on the configuration file as part of the documentation.
http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-Win-1.3.0/bk_installing_hdp_for_windows/content/win-getting-ready-6.html
You should make sure to have also installed the pre requisite.
Thanks
Olivier
On 3 Sep 2013 06:51, "Irfan Sayed" <irfu.sayed-***@public.gmane.org<mailto:***@gmail.com>> wrote:
thanks. sorry for the long break. actually got involved in some other priorities
i downloaded the installer and while installing i got following error
[Inline image 1]
do i need to make any configuration prior to installation ??
regards
irfan
On Fri, Aug 23, 2013 at 4:10 PM, Olivier Renault <orenault-RYNwJFaOa9CEK/***@public.gmane.org<mailto:orenault-RYNwJFaOa9CEK/***@public.gmane.org>> wrote:
Here is the link
http://download.hortonworks.com/products/hdp-windows/
Olivier
On 23 Aug 2013 10:55, "Irfan Sayed" <irfu.sayed-***@public.gmane.org<mailto:***@gmail.com>> wrote:
thanks.
i just followed the instructions to setup the pseudo distributed setup first using the url : http://cloudfront.blogspot.in/2012/07/how-to-configure-hadoop.html#.UhM8d2T0-4I
i don't think so i am running DN on both machine
please find the attached log
hi olivier
can you please give me download link ?
let me try please
regards
irfan
On Fri, Aug 23, 2013 at 1:08 PM, Mohammad Tariq <dontariq-***@public.gmane.org<mailto:dontariq-***@public.gmane.org>> wrote:
Are you running DN on both the machines? Could you please show me your DN logs?
Also, consider Oliver's suggestion. It's definitely a better option.
Warm Regards,
Tariq
cloudfront.blogspot.com<http://cloudfront.blogspot.com>
On Fri, Aug 23, 2013 at 12:57 PM, Olivier Renault <orenault-RYNwJFaOa9CEK/***@public.gmane.org<mailto:orenault-RYNwJFaOa9CEK/***@public.gmane.org>> wrote:
Irfu,
If you want to quickly get Hadoop running on windows platform. You may want to try our distribution for Windows. You will be able to find the msi on our website.
Regards
Olivier
On 23 Aug 2013 05:15, "Irfan Sayed" <irfu.sayed-***@public.gmane.org<mailto:***@gmail.com>> wrote:
thanks.
ok. i think i need to change the plan over here
let me create two environments. 1: totally windows 2: totally Unix
because, on windows , anyway i have to try and see how hadoop works
on UNIX, it is already known that , it is working fine.
so, on windows , here is the setup:
namenode : windows 2012 R2
datanode : windows 2012 R2
now, the exact problem is :
1: datanode is not getting started
2: replication : if i put any file/folder on any datanode , it should get replicated to all another available datanodes
regards
On Fri, Aug 23, 2013 at 2:42 AM, Mohammad Tariq <dontariq-***@public.gmane.org<mailto:dontariq-***@public.gmane.org>> wrote:
Seriously??You are planning to develop something using Hadoop on windows. Not a good idea. Anyways, cold you plz show me your log files?I also need some additional info :
-The exact problem which you are facing right now
-Your cluster summary(no. of nodes etc)
-Your latest configuration files
-Your /etc.hosts file
Warm Regards,
Tariq
cloudfront.blogspot.com<http://cloudfront.blogspot.com>
On Fri, Aug 23, 2013 at 1:42 AM, Irfan Sayed <irfu.sayed-***@public.gmane.org<mailto:irfu.sayed-***@public.gmane.org>> wrote:
ok. thanks
now, i need to start with all windows setup first as our product will be based on windows
so, now, please tell me how to resolve the issue
datanode is not starting . please suggest
regards,
irfan
On Thu, Aug 22, 2013 at 7:56 PM, Mohammad Tariq <dontariq-***@public.gmane.org<mailto:dontariq-***@public.gmane.org>> wrote:
It is possible. Theoretically Hadoop doesn't stop you from doing that. But it is not a very wise setup.
Warm Regards,
Tariq
cloudfront.blogspot.com<http://cloudfront.blogspot.com>
On Thu, Aug 22, 2013 at 5:01 PM, Irfan Sayed <irfu.sayed-***@public.gmane.org<mailto:irfu.sayed-***@public.gmane.org>> wrote:
please suggest
regards
irfan
On Thu, Aug 22, 2013 at 11:49 AM, Irfan Sayed <irfu.sayed-***@public.gmane.org<mailto:irfu.sayed-***@public.gmane.org>> wrote:
thanks.
can i have setup like this :
namenode will be on linux (flavour may be RHEL, CentOS, UBuntu etc)
and datanodes are the combination of any OS (windows , linux , unix etc )
however, my doubt is, as the file systems of both the systems (win and linux ) are different , datanodes of these systems can not be part of single cluster . i have to make windows cluster separate and UNIX cluster separate ?
regards
On Thu, Aug 22, 2013 at 11:26 AM, Arpit Agarwal <aagarwal-RYNwJFaOa9CEK/***@public.gmane.org<mailto:aagarwal-RYNwJFaOa9CEK/***@public.gmane.org>> wrote:
I just noticed you are on Cygwin. IIRC Windows PIDs are not the same as Cygwin PIDs so that may be causing the discrepancy. I don't know how well Hadoop works in Cygwin as I have never tried it. Work is in progress for native Windows support however there are no official releases with Windows support yet. It may be easier to get familiar with a release<https://www.apache.org/dyn/closer.cgi/hadoop/common/> on Linux if you are new to it.
On Wed, Aug 21, 2013 at 10:05 PM, Irfan Sayed <irfu.sayed-***@public.gmane.org<mailto:irfu.sayed-***@public.gmane.org>> wrote:
thanks
here is what i did .
i stopped all the namenodes and datanodes using ./stop-dfs.sh command
then deleted all pid files for namenodes and datanodes
started dfs again with command : "./start-dfs.sh"
when i ran the "Jps" command . it shows
***@DFS-DC /cygdrive/c/Java/jdk1.7.0_25/bin
$ ./jps.exe
4536 Jps
2076 NameNode
however, when i open the pid file for namenode then it is not showing pid as : 4560. on the contrary, it shud show : 2076
please suggest
regards
On Thu, Aug 22, 2013 at 9:59 AM, Arpit Agarwal <aagarwal-RYNwJFaOa9CEK/***@public.gmane.org<mailto:aagarwal-RYNwJFaOa9CEK/***@public.gmane.org>> wrote:
Most likely there is a stale pid file. Something like \tmp\hadoop-*datanode.pid. You could try deleting it and then restarting the datanode.
I haven't read the entire thread so you may have looked at this already.
-Arpit
On Wed, Aug 21, 2013 at 9:22 PM, Irfan Sayed <irfu.sayed-***@public.gmane.org<mailto:irfu.sayed-***@public.gmane.org>> wrote:
datanode is trying to connect to namenode continuously but fails
when i try to run "jps" command it says :
$ ./jps.exe
4584 NameNode
4016 Jps
and when i ran the "./start-dfs.sh" then it says :
$ ./start-dfs.sh
namenode running as process 3544. Stop it first.
DFS-1: datanode running as process 4076. Stop it first.
localhost: secondarynamenode running as process 4792. Stop it first.
both these logs are contradictory
please find the attached logs
should i attach the conf files as well ?
regards
On Wed, Aug 21, 2013 at 5:28 PM, Mohammad Tariq <dontariq-***@public.gmane.org<mailto:dontariq-***@public.gmane.org>> wrote:
Your DN is still not running. Showing me the logs would be helpful.
Warm Regards,
Tariq
cloudfront.blogspot.com<http://cloudfront.blogspot.com>
On Wed, Aug 21, 2013 at 5:11 PM, Irfan Sayed <irfu.sayed-***@public.gmane.org<mailto:irfu.sayed-***@public.gmane.org>> wrote:
i followed the url and did the steps mention in that. i have deployed on the windows platform
Now, i am able to browse url : http://localhost:50070 (name node )
however, not able to browse url : http://localhost:50030
please refer below
[Inline image 1]
i have modified all the config files as mentioned and formatted the hdfs file system as well
please suggest
regards
On Tue, Aug 20, 2013 at 4:14 PM, Irfan Sayed <irfu.sayed-***@public.gmane.org<mailto:irfu.sayed-***@public.gmane.org>> wrote:
thanks. i followed this url : http://blog.sqltrainer.com/2012/01/installing-and-configuring-apache.html
let me follow the url which you gave for pseudo distributed setup and then will switch to distributed mode
regards
irfan
On Tue, Aug 20, 2013 at 3:23 PM, Mohammad Tariq <dontariq-***@public.gmane.org<mailto:dontariq-***@public.gmane.org>> wrote:
You are welcome. Which link have you followed for the configuration?Your core-site.xml is empty. Remove the property fs.default.name<http://fs.default.name> from hdfs-site.xml and add it to core-site.xml. Remove mapred.job.tracker as well. It is required in mapred-site.xml.
I would suggest you to do a pseudo distributed setup first in order to get yourself familiar with the process and then proceed to the distributed mode. You can visit this link<http://cloudfront.blogspot.in/2012/07/how-to-configure-hadoop.html#.UhM8d2T0-4I> if you need some help. Let me know if you face any issue.
HTH
Warm Regards,
Tariq
cloudfront.blogspot.com<http://cloudfront.blogspot.com>
On Tue, Aug 20, 2013 at 2:56 PM, Irfan Sayed <irfu.sayed-***@public.gmane.org<mailto:irfu.sayed-***@public.gmane.org>> wrote:
thanks tariq for response.
as discussed last time, i have sent you all the config files in my setup .
can you please go through that ?
please let me know
regards
irfan
On Tue, Aug 20, 2013 at 1:22 PM, Mohammad Tariq <dontariq-***@public.gmane.org<mailto:dontariq-***@public.gmane.org>> wrote:
I'm sorry for being unresponsive. Was out of touch for sometime because of ramzan and eid. Resuming work today.
What's the current status?
Warm Regards,
Tariq
cloudfront.blogspot.com<http://cloudfront.blogspot.com>
On Mon, Aug 19, 2013 at 7:18 PM, manish dunani <manishd207-***@public.gmane.org<mailto:manishd207-***@public.gmane.org>> wrote:
First of all read the concepts ..I hope you will like it..
https://www.frcrc.org/sites/default/files/HadoopTutorialPart1.pdf
On Mon, Aug 19, 2013 at 9:45 AM, Irfan Sayed <irfu.sayed-***@public.gmane.org<mailto:irfu.sayed-***@public.gmane.org>> wrote:
please suggest
regards
irfan
On Tue, Aug 13, 2013 at 12:56 PM, Irfan Sayed <irfu.sayed-***@public.gmane.org<mailto:irfu.sayed-***@public.gmane.org>> wrote:
hey Tariq,
i am still stuck ..
can you please suggest
regards
irfan
On Thu, Aug 8, 2013 at 5:56 AM, Irfan Sayed <irfu.sayed-***@public.gmane.org<mailto:irfu.sayed-***@public.gmane.org>> wrote:
please suggest
regards
On Wed, Aug 7, 2013 at 9:49 AM, Irfan Sayed <irfu.sayed-***@public.gmane.org<mailto:irfu.sayed-***@public.gmane.org>> wrote:
attachment got quarantined
resending in txt format. please rename it to conf.rar
regards
On Wed, Aug 7, 2013 at 9:41 AM, Irfan Sayed <irfu.sayed-***@public.gmane.org<mailto:irfu.sayed-***@public.gmane.org>> wrote:
thanks.
if i run the jps command on namenode :
***@DFS-DC /cygdrive/c/Java/jdk1.7.0_25/bin
$ ./jps.exe
3164 NameNode
1892 Jps
same command on datanode :
***@DFS-1 /cygdrive/c/Java/jdk1.7.0_25/bin
$ ./jps.exe
3848 Jps
jps does not list any process for datanode. however, on web browser i can see one live data node
please find the attached conf rar file of namenode
regards
On Wed, Aug 7, 2013 at 1:52 AM, Mohammad Tariq <dontariq-***@public.gmane.org<mailto:dontariq-***@public.gmane.org>> wrote:
OK. we'll start fresh. Could you plz show me your latest config files?
BTW, are your daemons running fine?Use JPS to verify that.
Warm Regards,
Tariq
cloudfront.blogspot.com<http://cloudfront.blogspot.com>
On Tue, Aug 6, 2013 at 10:59 PM, Irfan Sayed <irfu.sayed-***@public.gmane.org<mailto:irfu.sayed-***@public.gmane.org>> wrote:
i have created these dir "wksp_data" and "wksp_name" on both datanode and namenode
made the respective changes in "hdfs-site.xml" file
formatted the namenode
started the dfs
but still, not able to browse the file system through web browser
please refer below
anything still missing ?
please suggest
[Inline image 1]
On Tue, Aug 6, 2013 at 10:35 PM, Irfan Sayed <irfu.sayed-***@public.gmane.org<mailto:irfu.sayed-***@public.gmane.org>> wrote:
these dir needs to be created on all datanodes and namenodes ?
further, hdfs-site.xml needs to be updated on both datanodes and namenodes for these new dir?
regards
On Tue, Aug 6, 2013 at 5:30 PM, Mohammad Tariq <dontariq-***@public.gmane.org<mailto:dontariq-***@public.gmane.org>> wrote:
Create 2 directories manually corresponding to the values of dfs.name.dir and dfs.data.dir properties and change the permissions of these directories to 755. When you start pushing data into your HDFS, data will start going inside the directory specified by dfs.data.dir and the associated metadata will go inside dfs.name.dir. Remember, you store data in HDFS, but it eventually gets stored in your local/native FS. But you cannot see this data directly on your local/native FS.
Warm Regards,
Tariq
cloudfront.blogspot.com<http://cloudfront.blogspot.com>
On Tue, Aug 6, 2013 at 5:26 PM, Irfan Sayed <irfu.sayed-***@public.gmane.org<mailto:irfu.sayed-***@public.gmane.org>> wrote:
thanks.
however, i need this to be working on windows environment as project requirement.
i will add/work on Linux later
so, now , at this stage , c:\\wksp is the HDFS file system OR do i need to create it from command line ?
please suggest
regards,
On Tue, Aug 6, 2013 at 5:19 PM, Mohammad Tariq <dontariq-***@public.gmane.org<mailto:dontariq-***@public.gmane.org>> wrote:
Hello Irfan,
Sorry for being unresponsive. Got stuck with some imp work.
HDFS webUI doesn't provide us the ability to create file or directory. You can browse HDFS, view files, download files etc. But operation like create, move, copy etc are not supported.
These values look fine to me.
One suggestion though. Try getting a Linux machine(if possible). Or at least use a VM. I personally feel that using Hadoop on windows is always messy.
Warm Regards,
Tariq
cloudfront.blogspot.com<http://cloudfront.blogspot.com>
On Tue, Aug 6, 2013 at 5:09 PM, Irfan Sayed <irfu.sayed-***@public.gmane.org<mailto:irfu.sayed-***@public.gmane.org>> wrote:
thanks.
when i browse the file system , i am getting following :
i haven't seen any make directory option there
i need to create it from command line ?
further, in the hdfs-site.xml file , i have given following entries. are they correct ?
<property>
<name>dfs.data.dir</name>
<value>c:\\wksp</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>c:\\wksp</value>
</property>
please suggest
[Inline image 1]
On Tue, Aug 6, 2013 at 12:40 PM, manish dunani <manishd207-***@public.gmane.org<mailto:manishd207-***@public.gmane.org>> wrote:
You are wrong at this:
***@DFS-DC /cygdrive/c/hadoop-1.1.2/hadoop-1.1.2/bin
$ ./hadoop dfs -copyFromLocal /cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar /wksp
copyFromLocal: File /cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar does not exist.
***@DFS-DC /cygdrive/c/hadoop-1.1.2/hadoop-1.1.2/bin
$ ./hadoop dfs -copyFromLocal /cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar.gz /wksp
copyFromLocal: File /cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar.gz does not exist.
Because,You had wrote both the paths local and You need not to copy hadoop into hdfs...Hadoop is already working..
Just check out in browser by after starting ur single node cluster :
localhost:50070
then go for browse the filesystem link in it..
If there is no directory then make directory there.
That is your hdfs directory.
Then copy any text file there(no need to copy hadoop there).beacause u are going to do processing on that data in text file.That's why hadoop is used for ,first u need to make it clear in ur mind.Then and then u will do it...otherwise not possible..
Try this:
***@DFS-DC /cygdrive/c/hadoop-1.1.2/hadoop-1.1.2
$ .bin/hadoop dfs -copyFromLocal /full/local/path/to/ur/file /hdfs/directory/path
On Tue, Aug 6, 2013 at 11:49 AM, Irfan Sayed <irfu.sayed-***@public.gmane.org<mailto:irfu.sayed-***@public.gmane.org>> wrote:
thanks. yes , i am newbie.
however, i need windows setup.
let me surely refer the doc and link which u sent but i need this to be working ...
can you please help
regards
--
MANISH DUNANI
-THANX
+91 9426881954<tel:%2B91%209426881954>,+91 8460656443<tel:%2B91%208460656443>
manishd207-***@public.gmane.org<mailto:manishd207-***@public.gmane.org>
--
Regards
Manish Dunani
Contact No : +91 9408329137<tel:%2B91%209408329137>
skype id : manish.dunani
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
--
Olivier Renault
Solution Engineer - Big Data - Hortonworks, Inc.
+44 7500 933 036
orenault-RYNwJFaOa9CEK/***@public.gmane.org<mailto:orenault-RYNwJFaOa9CEK/***@public.gmane.org>
www.hortonworks.com<http://www.hortonworks.com/>
<http://hortonworks.com/products/hortonworks-sandbox/>
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.<http://hortonworks.com/products/hortonworks-sandbox/>
<http://hortonworks.com/products/hortonworks-sandbox/>