newbie question on disk usage on node with different disk size

[Tux Racer]

Hello Hbase Users!

I am trying to find some pointers on how to configure hbase region
server and in particular how the disk will be filled on each node.

Say for instance that I have a small cluster of 3 nodes:
node 1 has a 100Gb disk
node 2 has a 200Gb disk
and node 3 has a 300 Gb disk

is there a way to tell hbase that it should store the keys
proportionally to the he node disk space?
(i.e. to have at some stage each disk filled at 50%: 50/100/150 Gb of
space used)

Or is that a pure Hadoop configuration question?
I looked at the files in the ~/hbase-0.20.1/conf/ folder with no luck.

Thanks
TR

[Andrew Purtell]

Hi,

Short answer: No.

Longer answer: HBase uses the underlying filesystem (typically HDFS) to replicate and persist data. This is independent of the key space. Any special block placement policy like you want would be handled by the filesystem. To my knowledge, HDFS doesn't support it. HDFS also does not like heterogeneous backing storage at the moment. It causes problems if one node fills before the others, and there is not yet an automatic mechanism for moving blocks from full nodes to less utilized ones, though I see there is an issue for that: http://issues.apache.org/jira/browse/HDFS-339 . I wouldn't recommend a setup like you propose.

- Andy



________________________________
From: Tux Racer <tuxracer69-***@public.gmane.org>
To: hbase-user-7ArZoLwFLBtd/SJB6HiN2Ni2O/***@public.gmane.org
Sent: Thu, November 26, 2009 11:14:15 AM
Subject: newbie question on disk usage on node with different disk size

Hello Hbase Users!

I am trying to find some pointers on how to configure hbase region server and in particular how the disk will be filled on each node.

Say for instance that I have a small cluster of 3 nodes:
node 1 has a 100Gb disk
node 2 has a 200Gb disk
and node 3 has a 300 Gb disk

is there a way to tell hbase that it should store the keys proportionally to the he node disk space?
(i.e. to have at some stage each disk filled at 50%: 50/100/150 Gb of space used)

Or is that a pure Hadoop configuration question?
I looked at the files in the ~/hbase-0.20.1/conf/ folder with no luck.

Thanks
TR

[Tux Racer]

Thanks Andrew for your answer.

I may have found the HDFS parameter I was looking for:

dfs.datanode.du.reserved 0 Reserved space in bytes per volume. Always
leave this much space free for non dfs use.


http://hadoop.apache.org/common/docs/current/hdfs-default.html
https://issues.apache.org/jira/browse/HADOOP-1463

The book "Pro Hadoop" page 115, also mentions the "balancer" service and
the

dfs.balance.bandwidthPerSec

parameter also documented at:

http://hadoop.apache.org/common/docs/current/hdfs-default.html

dfs.balance.bandwidthPerSec 1048576 Specifies the maximum amount of
bandwidth that each datanode can utilize for the balancing purpose in
term of the number of bytes per second.



however I do not see the script "start-blancer.sh" in hbase.

Would it be possible to use those Hadoop parameters in a hbase setup?

Thanks
TR

Post by Andrew Purtell
Hi,
Short answer: No.
Longer answer: HBase uses the underlying filesystem (typically HDFS) to replicate and persist data. This is independent of the key space. Any special block placement policy like you want would be handled by the filesystem. To my knowledge, HDFS doesn't support it. HDFS also does not like heterogeneous backing storage at the moment. It causes problems if one node fills before the others, and there is not yet an automatic mechanism for moving blocks from full nodes to less utilized ones, though I see there is an issue for that: http://issues.apache.org/jira/browse/HDFS-339 . I wouldn't recommend a setup like you propose.
- Andy
________________________________
Sent: Thu, November 26, 2009 11:14:15 AM
Subject: newbie question on disk usage on node with different disk size
Hello Hbase Users!
I am trying to find some pointers on how to configure hbase region server and in particular how the disk will be filled on each node.
node 1 has a 100Gb disk
node 2 has a 200Gb disk
and node 3 has a 300 Gb disk
is there a way to tell hbase that it should store the keys proportionally to the he node disk space?
(i.e. to have at some stage each disk filled at 50%: 50/100/150 Gb of space used)
Or is that a pure Hadoop configuration question?
I looked at the files in the ~/hbase-0.20.1/conf/ folder with no luck.
Thanks
TR

[Ryan Rawson]

I think you might be mistaken a bit - HBase layers on top of, and uses
hadoop. HBase uses HDFS for persistence, and thus the balancer config
and the other things you point out belong in the hadoop config.

3 nodes is a little light for HDFS... With r=3, there is are no spares.

Post by Tux Racer
Thanks Andrew for your answer.
dfs.datanode.du.reserved        0       Reserved space in bytes per volume.
Always leave this much space free for non dfs use.
http://hadoop.apache.org/common/docs/current/hdfs-default.html
https://issues.apache.org/jira/browse/HADOOP-1463
The book "Pro Hadoop" page 115, also mentions the "balancer" service and the
dfs.balance.bandwidthPerSec
http://hadoop.apache.org/common/docs/current/hdfs-default.html
dfs.balance.bandwidthPerSec     1048576         Specifies the maximum amount
of bandwidth that each datanode can utilize for the balancing purpose in
term of the number of bytes per second.
however I do not see the script "start-blancer.sh" in hbase.
Would it be possible to use those Hadoop parameters in a hbase setup?
Thanks
TR

Post by Andrew Purtell
Hi,
Short answer: No.
Longer answer: HBase uses the underlying filesystem (typically HDFS) to
replicate and persist data. This is independent of the key space. Any
special block placement policy like you want would be handled by the
filesystem. To my knowledge, HDFS doesn't support it. HDFS also does not
like heterogeneous backing storage at the moment. It causes problems if one
node fills before the others, and there is not yet an automatic mechanism
for moving blocks from full nodes to less utilized ones, though I see there
is an issue for that: http://issues.apache.org/jira/browse/HDFS-339 . I
wouldn't recommend a setup like you propose.
  - Andy
________________________________
Sent: Thu, November 26, 2009 11:14:15 AM
Subject: newbie question on disk usage on node with different disk size
Hello Hbase Users!
I am trying to find some pointers on how to configure hbase region server
and in particular how the disk will be filled on each node.
node 1 has a 100Gb disk
node 2 has a 200Gb disk
and node 3 has a 300 Gb disk
is there a way to tell hbase that it should store the keys proportionally
to the he node disk space?
(i.e. to have at some stage each disk filled at 50%: 50/100/150 Gb of space used)
Or is that a pure Hadoop configuration question?
I looked at the files in the ~/hbase-0.20.1/conf/ folder with no luck.
Thanks
TR

[Tux Racer]

Thanks Ryan for your answer.
yes I was mistaken, I also thought that the default install of hbase did
a one node install of HDFS; and it seems that wrong:

a ps auwx|grep java

show only two java processes;

org.apache.hadoop.hbase.zookeeper.HQuorumPeer
and
org.apache.hadoop.hbase.master.HMaster

In the default hbase distribution we have in

hbase-default.xml

<name>hbase.rootdir</name>
<value>file:///tmp/hbase-${user.name}/hbase</value>

I thought that the dependancy of hbase on HDFS was much stronger. For
the hbase configuration point of view if the hbase.rootdir parameter the
only parameter that hooks hbase to HDFS?
Or does zookeeper also binds hbase to HDFS?
Is it true to say that hbase does play well with HDFS but that it does
play well with any POSIX compliant filesystem too?
For a small cluster, is that a good idea to *not* use HDFS as a storage
for the hbase data?
If I accept to loose one hour of hbase data, is it OK to make
hbase.rootdir point to local (ext3) file system on the node and then
rsync each hour that directory to another node? I guess that rsync is
not ideal due to the file structure used (will generate a lot of network
traffic)

Thanks in advance,
TR

Post by Ryan Rawson
I think you might be mistaken a bit - HBase layers on top of, and uses
hadoop. HBase uses HDFS for persistence, and thus the balancer config
and the other things you point out belong in the hadoop config.
3 nodes is a little light for HDFS... With r=3, there is are no spares.

[Ryan Rawson]

Hey,

When you start hbase in a fresh installation it will use local fs in
/tmp. The hadoop filesystem libraries we use allow the use of at
least 3 filesystems (local, hdfs, kfs). Right now you are seeing the
single ZK process and the combined HBase master/regionserver process.

HBase needs the following things out of it's filesystem:
- global view - every single regionserver & master MUST see every file
from everyone at all times. 1 hour rsync won't cut it.
- high bandwidth, once you get 3+ servers doing high IO (compaction,
etc), you wont want to rely on a 1 disk NFS.

In theory you can use something like NFS and common mount dir on all
regionservers/masters. This won't scale of course. It should _in
theory_ work... You can specify the rootdir with something like
"file:///nfs_mount_path/hbase". Normally we'd say
hdfs://namenode:port/hbase

The hbase scripts don't boot up or control hadoop at all. You must
provide a working hadoop, then hbase can use it. It may seem a little
"annoying" to have a 2 step process, but the decoupled control makes
our control scripts more generic and suitable for all.

Good luck out there!
-ryan

Post by Tux Racer
Thanks Ryan for your answer.
yes I was mistaken, I also thought that the default install of hbase did a
a ps auwx|grep java
show only two java processes;
org.apache.hadoop.hbase.zookeeper.HQuorumPeer
and
org.apache.hadoop.hbase.master.HMaster
In the default hbase distribution we have in
hbase-default.xml
<name>hbase.rootdir</name>
<value>file:///tmp/hbase-${user.name}/hbase</value>
I thought that the dependancy of hbase on HDFS was much stronger. For the
hbase configuration point of view if the hbase.rootdir parameter the only
parameter that hooks hbase to HDFS?
Or does zookeeper also binds hbase to HDFS?
Is it true to say that hbase does play well with HDFS but that it does play
well with any POSIX compliant filesystem too?
For a small cluster, is that a good idea to *not* use HDFS as a storage for
the hbase data?
If I accept to loose one hour of hbase data, is it OK to make hbase.rootdir
point to  local (ext3) file system on the node and then rsync each hour that
directory to another node? I guess that rsync is not ideal due to the file
structure used (will generate a lot of network traffic)
Thanks in advance,
TR

Post by Ryan Rawson
I think you might be mistaken a bit - HBase layers on top of, and uses
hadoop.  HBase uses HDFS for persistence, and thus the balancer config
and the other things you point out belong in the hadoop config.
3 nodes is a little light for HDFS... With r=3, there is are no spares.

[Tux Racer]

Ryan:
actually I really like this decoupling Hbase/filesystem :)
Thanks again all for your replies
Cheers, TR

Post by Ryan Rawson
Hey,
When you start hbase in a fresh installation it will use local fs in
/tmp. The hadoop filesystem libraries we use allow the use of at
least 3 filesystems (local, hdfs, kfs). Right now you are seeing the
single ZK process and the combined HBase master/regionserver process.
- global view - every single regionserver & master MUST see every file
from everyone at all times. 1 hour rsync won't cut it.
- high bandwidth, once you get 3+ servers doing high IO (compaction,
etc), you wont want to rely on a 1 disk NFS.
In theory you can use something like NFS and common mount dir on all
regionservers/masters. This won't scale of course. It should _in
theory_ work... You can specify the rootdir with something like
"file:///nfs_mount_path/hbase". Normally we'd say
hdfs://namenode:port/hbase
The hbase scripts don't boot up or control hadoop at all. You must
provide a working hadoop, then hbase can use it. It may seem a little
"annoying" to have a 2 step process, but the decoupled control makes
our control scripts more generic and suitable for all.
Good luck out there!
-ryan

[Andrew Purtell]

The balancer moves blocks so they are evenly distributed across the cluster. It must be run manually and won't do what you want once the datanode with the smallest disk fills.

Neither the HDFS balancer nor the config var you mention is part of HBase. You can run the balancer with the Hadoop scripts and specify a change to that config var in Hadoop's hdfs-site.xml.

- Andy

Post by Tux Racer
Thanks Andrew for your answer.
dfs.datanode.du.reserved 0 Reserved space in bytes per volume. Always
leave this much space free for non dfs use.
http://hadoop.apache.org/common/docs/current/hdfs-default.html
https://issues.apache.org/jira/browse/HADOOP-1463
The book "Pro Hadoop" page 115, also mentions the "balancer" service and
the
dfs.balance.bandwidthPerSec
http://hadoop.apache.org/common/docs/current/hdfs-default.html
dfs.balance.bandwidthPerSec 1048576 Specifies the maximum amount of
bandwidth that each datanode can utilize for the balancing purpose in
term of the number of bytes per second.
however I do not see the script "start-blancer.sh" in hbase.
Would it be possible to use those Hadoop parameters in a hbase setup?
Thanks
TR

Post by Andrew Purtell
Hi,
Short answer: No.
Longer answer: HBase uses the underlying filesystem (typically HDFS) to replicate and persist data. This is independent of the key space. Any special block placement policy like you want would be handled by the filesystem. To my knowledge, HDFS doesn't support it. HDFS also does not like heterogeneous backing storage at the moment. It causes problems if one node fills before the others, and there is not yet an automatic mechanism for moving blocks from full nodes to less utilized ones, though I see there is an issue for that: http://issues.apache.org/jira/browse/HDFS-339 . I wouldn't recommend a setup like you propose.
- Andy
________________________________
Sent: Thu, November 26, 2009 11:14:15 AM
Subject: newbie question on disk usage on node with different disk size
Hello Hbase Users!
I am trying to find some pointers on how to configure hbase region server and in particular how the disk will be filled on each node.
node 1 has a 100Gb disk
node 2 has a 200Gb disk
and node 3 has a 300 Gb disk
is there a way to tell hbase that it should store the keys proportionally to the he node disk space?
(i.e. to have at some stage each disk filled at 50%: 50/100/150 Gb of space used)
Or is that a pure Hadoop configuration question?
I looked at the files in the ~/hbase-0.20.1/conf/ folder with no luck.
Thanks
TR