Discussion:
[gentoo-user] Rasp-Pi-4 Gentoo servers
(too old to reply)
james
2020-02-27 04:50:01 UTC
Permalink
I'm thinking about setting up a pair of Rasp-Pi-4 as DNS servers with
4GB of ram. Is that enough ram for a DNS server?

https://www.amazon.com/CanaKit-Raspberry-4GB-Starter-MAX/dp/B07XPHWPRB


https://wiki.gentoo.org/wiki/Embedded_systems/ARM_hardware_list

If 4GB is not enough, there are some boards with 8GB
of ram.


What about running postfix on a third board?

https://samhobbs.co.uk/2013/12/raspberry-pi-email-server-part-1-postfix


All feedback is encouraged. It will not be a high volume postfix email
system. As far as the DNS servers, it seems I vaguely remember some
software or filters to prevent hack attacks, that just overwhelm DNS
servers; or something like that. It just lowers the data flow rate.

It seems this is popular, and which version of USB 3.?
to use on a stick, as now usb sticks can have large capacities now (T+).

Surely I'll back up the mail (spool & such) to a secondary hard drive on
another system.

It this all goes well, surely I put a web server on a fourth board.

https://pimylifeup.com/raspberry-pi-nginx/


Granted those links are not centric to embedded gentoo, but they do
cover a lot of what is needed.

Further suggestions are most welcome. This is a fun to do project,
useful as it would basically be easy to duplicate, so I can travel
anywhere, get a few static IPs and deploy a small business,
self-sufficient, network.

So the Rpi4 boards will all need their own unique, static Ip. But,
another questions is while they are running as 4 distinct gentoo
servers, could they also double as a quad-gentoo cluster?

https://www.picocluster.com/collections/raspberry-pi4


Security pointers are most welcome too.
The eventual idea is to have a fixed home network, but
be able to travel around in my pickup/cabovercamper
and have what I need right there, or provide it to friends as a ready to
go small network. Embedded Gentoo,
low-power hardware and 100% (gentoo) source driven).

The S20 (520G ram) phone would server as the router.
(2) 5G cell phones running on (2) different 5G service
vendor, could make the services and cluster multi-homed?

Would all of this work with just IP6 and a fancy cell phone, running
gentoo? We shall see.


James
Ralph Seichter
2020-02-27 15:20:01 UTC
Permalink
Post by james
I'm thinking about setting up a pair of Rasp-Pi-4 as DNS servers with
4GB of ram. Is that enough ram for a DNS server?
For running the Nameservers, yes. Compiling Gentoo packages will likely
put your SD-Card under stress, but that's just how it goes. My Model B
Rev 2 of 2015 runs dnsmasq as DHCP server, NGINX, Postfix, Unbound and
more for a bunch of clients in a LAN. It is quite nifty as a local DNS
Resolver and DHCP server, because it is usually the fastest to boot
after the occasional power outage.

I would not use it as an Internet-facing production Mailserver, though,
because that would generate a lot of I/O, which is not a Raspberry Pi
strong suit.

-Ralph
aisha
2020-02-27 19:20:01 UTC
Permalink
I'm not too sure that running it as a mail server is impossible.
Depending on your expected traffic level, it should be more than capable
enough to do it.
My current server is only a 1 core + 1 GB VPS, which is much more lax
than a pi-4.

Depending on what guides you follow you can definitely set it up as a
mail server.

But I am curious how you are planning to do this, unless you have a
static ip + reverse DNS configured?


---
Aisha
blog.aisha.cc
Post by Ralph Seichter
Post by james
I'm thinking about setting up a pair of Rasp-Pi-4 as DNS servers with
4GB of ram. Is that enough ram for a DNS server?
For running the Nameservers, yes. Compiling Gentoo packages will likely
put your SD-Card under stress, but that's just how it goes. My Model B
Rev 2 of 2015 runs dnsmasq as DHCP server, NGINX, Postfix, Unbound and
more for a bunch of clients in a LAN. It is quite nifty as a local DNS
Resolver and DHCP server, because it is usually the fastest to boot
after the occasional power outage.
I would not use it as an Internet-facing production Mailserver, though,
because that would generate a lot of I/O, which is not a Raspberry Pi
strong suit.
-Ralph
Ralph Seichter
2020-02-27 20:00:02 UTC
Permalink
Post by aisha
I'm not too sure that running it as a mail server is impossible.
I never wrote that it is impossible, only that "I would not use it as an
Internet-facing production Mailserver". That's a huge difference. You
are free to do as you wish, but I still consider it an unsuitable role
for a wee Rasberry Pi, considering the I/O load I see on our production
mail servers. SD-Cards really don't like this sort of thing.

-Ralph
james
2020-02-27 21:30:02 UTC
Permalink
Post by Ralph Seichter
Post by aisha
I'm not too sure that running it as a mail server is impossible.
I never wrote that it is impossible, only that "I would not use it as an
Internet-facing production Mailserver". That's a huge difference. You
are free to do as you wish, but I still consider it an unsuitable role
for a wee Rasberry Pi, considering the I/O load I see on our production
mail servers. SD-Cards really don't like this sort of thing.
-Ralph
Yea, I was not clear. I'd run the mail-server, on a 'cluster' (4 or
more), not an individual pi-board unless it was beef up, processor and
ram wise. Gig E would also be on my list.

There are also embedded boards, that can run gentoo, with up to 16 Gigs
of DDR4 ram and better internal hardware for threads and such.....

I certainly, did not mean to offend you, so apologies galore. I'm not
into running a mail server for more than a dozen folks and an ity-bity
company of just one....

SD card? I'd find an embedded-board that runs (8Gbyte)DDR4 ram, so
writes to the storage is very fast. It's a bit too detailed to look at
the plethora of hardware available, that one can get for embedded
projects and the matching (sensitive) price points. No need to go there
(its a morass).


thanks,
James
Rich Freeman
2020-02-27 22:00:02 UTC
Permalink
Post by james
Yea, I was not clear. I'd run the mail-server, on a 'cluster' (4 or
more), not an individual pi-board unless it was beef up, processor and
ram wise. Gig E would also be on my list.
Unless you have some niche need I wouldn't generally run servers on
Pis. The biggest issue with ARM is that all the cheap platforms are
starved for RAM, and RAM is one of the biggest issues when running
services. And of course the Pi in particular has IO issues (as do
many other cheap SBCs but this is less of an ARM issue). The RAM
issue isn't so many an ARM issue as a supply/demand thing - the only
people asking for 64GB ARM boards are big companies that are willing
to pay a lot for them.

I do actually run a few services on Pis - DNS, DHCP, and a VPN
gateway. That's about it. These are fairly non-demanding tasks that
the hardware doesn't struggle with, and the data is almost entirely
static so an occasional backup makes any kind of recovery trivial.
The only reason I run these services on Pis is that they are fairly
fundamental to having a working network. Most of my services are
running in containers on a server, but I don't want to have to think
about taking a server down for maintenance and then literally every
IOT device in the house won't work. These particular services are
also basically dependency-free which means I can just boot them up and
they just do their jobs, while they remain a dependency for just about
everything else on the network. When you start running DHCP in a
container you have more complex dependency issues.

A fairly cheap amd64 system can run a ton of services in containers
though, and it is way simpler to maintain that way. I still get quick
access to snapshots/etc, but now if I want to run a gentoo container
it is no big deal if 99% of the time it uses 25MB of RAM and 1% of one
core, but once a month it needs 4GB of RAM and 100% of 6 cores. As
long as I'm not doing an emerge -u world on half a dozen containers at
once it is no big deal at all.

Now, if I needed some server in some niche application that needed to
be able to operate off of a car battery for a few days, then sure I'd
be looking at Pis and so on.
--
Rich
james
2020-02-27 23:10:01 UTC
Permalink
Post by Rich Freeman
Post by james
Yea, I was not clear. I'd run the mail-server, on a 'cluster' (4 or
more), not an individual pi-board unless it was beef up, processor and
ram wise. Gig E would also be on my list.
Unless you have some niche need I wouldn't generally run servers on
Pis. The biggest issue with ARM is that all the cheap platforms are
starved for RAM, and RAM is one of the biggest issues when running
services. And of course the Pi in particular has IO issues (as do
many other cheap SBCs but this is less of an ARM issue). The RAM
issue isn't so many an ARM issue as a supply/demand thing - the only
people asking for 64GB ARM boards are big companies that are willing
to pay a lot for them.
I do actually run a few services on Pis - DNS, DHCP, and a VPN
gateway. That's about it. These are fairly non-demanding tasks that
the hardware doesn't struggle with, and the data is almost entirely
static so an occasional backup makes any kind of recovery trivial.
The only reason I run these services on Pis is that they are fairly
fundamental to having a working network. Most of my services are
running in containers on a server, but I don't want to have to think
about taking a server down for maintenance and then literally every
IOT device in the house won't work. These particular services are
also basically dependency-free which means I can just boot them up and
they just do their jobs, while they remain a dependency for just about
everything else on the network. When you start running DHCP in a
container you have more complex dependency issues.
A fairly cheap amd64 system can run a ton of services in containers
though, and it is way simpler to maintain that way. I still get quick
access to snapshots/etc, but now if I want to run a gentoo container
it is no big deal if 99% of the time it uses 25MB of RAM and 1% of one
core, but once a month it needs 4GB of RAM and 100% of 6 cores. As
long as I'm not doing an emerge -u world on half a dozen containers at
once it is no big deal at all.
Now, if I needed some server in some niche application that needed to
be able to operate off of a car battery for a few days, then sure I'd
be looking at Pis and so on.
Exactly. It's going to be a small RV, basically a 4x4 with a
campershell. 2 Laptops with AMD64 and ram, the newest ones, are the
powerhouses. One multicore with Radeon Graphics with a stand for 4x32"
4K 120MHZ screens to be mounted on the 1/2 table in front of the bench
seat (my new mobile office). I'm hoping to get gcc-9 (10?) happy with
auto-compiling using the AMD graphics (radeon chipsets). And a plethora
of small embedded boards for a wide variety of toy-interfaces.....


Biggest problem? The arrival of the new roof mounted, 12VDC 21 SEER AC
is delayed, due to that virus. Trying not to crank a genset, just solar,
and fast-charge 12VDC battery banks; but we'll see how that goes in the
Texas summer. I'm going to map out some 5G hot-spots and encourage folks
in the areas to jump on a gentoo (derivative?) OS. The greater Dallas
area is a hotbed for 5G testing and development. Austin is on fire with
tons of new technologies too. Texas is pumping serious monies into
everything 5G. It'll be an addon package for guys with tractors......


Making all of this fun and easy with Gentoo, should help grow our
distro. The Texas Universities are moving to a new multi-homed private
fiber network, where each link is 100G fiber based. So every campus in
Texas, will soon be hotbeds for 5G R&D and play. Since the state of
Texas has many 5G chip manufactures and many custom Rf shops, it's gonna
be the worlds hotspot for 5G, imho. There's talk of Texas sharing this
100G multi-route network with Oklahoma, Arkansas and Mississippi; a
brilliant move imho. Gentoo could 'own' Texas, with just a wee bit of
effort. IBM taking over CoreOS(new version of RedHat) is leaving a very
foul taste in many circles.....


5G + gentoo + embedded toys, is going to be FUN FUN FUN.


Then I'll be off to other states, via a hacked out Redneck camper......
and too many microProcessors....


Thanks Rich, your insights and comments are always most welcome.


James
Dale
2020-02-28 03:00:01 UTC
Permalink
Post by james
5G + gentoo + embedded toys, is going to be FUN FUN FUN.
Then I'll be off to other states, via a hacked out Redneck
camper...... and too many microProcessors....
Thanks Rich, your insights and comments are always most welcome.
James
Off topic a bit but a question.  Would one of these Rasp-Pi-4 thingys
make a NAS hard drive server? I have a Cooler Master HAF-932 case but
even it is running out of hard drive space.  I'm thinking about building
a NAS box, taking sheet metal and bending it until it looks like a box. 
Thing is, it needs a small puter to take data from the drives to the
network and vice-versa. 

I've never even seen one of those things, except on my monitor, so I
have no idea what all they are capable of.  I figure a lot of SATA
connectors and a ethernet connection plus enough CPU power and memory to
get the job done.  If those things are capable of doing that fairly
easily.  After all, I'm me.  :/

Just curious.

Dale

:-)  :-) 
james
2020-02-28 05:10:01 UTC
Permalink
Post by james
5G + gentoo + embedded toys, is going to be FUN FUN FUN.
Then I'll be off to other states, via a hacked out Redneck
camper...... and too many microProcessors....
Thanks Rich, your insights and comments are always most welcome.
James
Off topic a bit but a question.� Would one of these Rasp-Pi-4 thingys
make a NAS hard drive server?
Sure, but, there may be a better solution, something all ready out there
and it really depends on refining your needs, current and in the future.
So lets refine your specifications (centric to your needs + growth) and
figure out what and how much you need. Then we can survey the
embedded-thingies, that meet your specs, with a bit of room for growth, OK?
I have a Cooler Master HAF-932 case
Wow, that's big. What the number and capacity (TB) of
your existing hard-drives?

How much more storage do you want? Replacing drives with larger
capacity, might be all you need to do?
but
even it is running out of hard drive space.� I'm thinking about building
a NAS box, taking sheet metal and bending it until it looks like a box.
OK, so we first spec out options, then let you decide. Then you can
'bargain shop' for appropriate housing/rack/open chassis, etc.
Thing is, it needs a small puter to take data from the drives to the
network and vice-versa.
embedded are not only small, they can have extended temperature ranges
of tolerance, use drastically less power and many other features. If
it's purposed hardware, that is only a few things todo, then yes
embedded uP (abbrev for microProcessor) are the way to go. Running off
of 12VDC, means an old car battery and a connection to your solar panels
(assuming you have those) and it's zero on your electric bill. There is
usually a vast array of tax and other incentives, particularly with
solar in Ag businesses.
I've never even seen one of those things, except on my monitor, so I
have no idea what all they are capable of.
Dale, you are pretty strong with Gentoo Linux, so putting a stripped,
purposed, minimized gentoo derivative stack, with far less ebuilds, to
work for your operations, is going to be quite fun. On a farm or ranch,
there are a myriad of things you can do with embedded boards and
gentoo-stripped. You can replace many of those expensive (vendor)
systems with embedded boards +sensors +controls codes and lots of wires
to do most anything. Let's focus on your NAS for now.
I figure a lot of SATA connectors and a ethernet connection
plus enough CPU power and memory to get the job done.
SATA, was great years ago. Still it makes sense to use, if you already
have them. Storage going forward is the process of faster and cheaper
and leaving SATA behind, like ide. Still useful, but a power hog. So
we'll start out with interfacing your existing SATA drives to the
embedded board, and look/decide on options for newer Solid State Data
storage options.


� https://en.wikipedia.org/wiki/USB

You might not even need many sata ports. usb3 and the upcoming usb4 have
tons of bandwidth (date/time). Mechanical Hard drives are on the way
out. Too expensive and failure prone. SSd and other types of storage,
might be right for you, or a mixture. USB stick memory
can be huge, very low power draw and very inexpensive.

A hybrid of several types of memory storage may be useful to experiment
with. You may want to categorize your long term storage: some accessed
often, others maybe once a year?


For data storage, long term important stuff, you should employ RAID
(1-10). We can get into that later, duplication of important data, via
backups or extra storage is a good idea too. Backups are an old
technology, but may help, but backups do can get old too and fragmented.
For now, lets not worry so much about long term bit integrity, but focus
on your next FUN gentoo rig. I'm hoping other join in to so you have
more than my prospective on your solution.
If those things are capable of doing that fairly
easily.� After all, I'm me.� :/
OK, so let's survey some system, you can just purchase
with gentoo preinstalled, or a very easy pathway to embedded gentoo.
Let's look at a few, have some of the other guys jump in, and find you a
solution, to start with. Most will be expandable, and you can figure out
the casing, mounting, power and such.

At this stage, it mostly a research effort and then deciding your
features/price. If you do not have massive bandwitdh requirements, I'm
sure we can find you
a very cost effective, DC powered solution.

Just so you know, I use that fancy $300 OPtima 12vdc charger, and Optima
batteries. the charger reconditions most batteries, if they are not
beyond saving, even cheap lead-acid batteries. Every Farmer should have
one, imho. The Digital 1200 is just awesome.

https://www.optimabatteries.com/en-us/battery-charger

If you like, you can read up on blue, red and yellow top versions and
their intended duty-cycle.

Take some time and look at these gentoo embedded boards, before we
diverge to other offerings:

https://wiki.gentoo.org/wiki/Embedded_systems/ARM_hardware_list


https://www.raspberrypi.org/products/raspberry-pi-4-model-b/
Just curious.
Dale
:-)� :-)
Dale, embedded gentoo is about the most fun thing you'll ever do. A
hardware guy like yourself, that is already an accomplish part of the
great gentoo community, dude,
your going to have a lot of fun, and save money on that farm. So let's
get started!

Post back and I, or others, will try to answer your questions. Be
specific about what you do currently and where you want to end up.

Just so you know, imho, (2) of the finest humans and accomplished Gentoo
devs are listed in gentoo-embedded.
Actually, all the members are just totally awesome humans, so you, my
friend, are in for the ride of a lifetime!


https://wiki.gentoo.org/wiki/Project:Embedded


https://developer.arm.com/tools-and-software/open-source-software/developer-tools/gnu-toolchain/gnu-rm

More boards to look at (the features).

https://www.embeddedarm.com/products/category/single-board-computers

https://freaklabs.org/chibiarduino/

https://www.reddit.com/r/Gentoo/comments/7oxoln/question_about_gentoo_on_raspberry_pi/
Michael
2020-02-28 10:50:01 UTC
Permalink
Post by james
Post by james
5G + gentoo + embedded toys, is going to be FUN FUN FUN.
Then I'll be off to other states, via a hacked out Redneck
camper...... and too many microProcessors....
Thanks Rich, your insights and comments are always most welcome.
James
Off topic a bit but a question.ᅵ Would one of these Rasp-Pi-4 thingys
make a NAS hard drive server?
Sure, but, there may be a better solution, something all ready out there
and it really depends on refining your needs, current and in the future.
So lets refine your specifications (centric to your needs + growth) and
figure out what and how much you need. Then we can survey the
embedded-thingies, that meet your specs, with a bit of room for growth, OK?
I have a Cooler Master HAF-932 case
Wow, that's big. What the number and capacity (TB) of
your existing hard-drives?
How much more storage do you want? Replacing drives with larger
capacity, might be all you need to do?
but
even it is running out of hard drive space.ᅵ I'm thinking about building
a NAS box, taking sheet metal and bending it until it looks like a box.
OK, so we first spec out options, then let you decide. Then you can
'bargain shop' for appropriate housing/rack/open chassis, etc.
Thing is, it needs a small puter to take data from the drives to the
network and vice-versa.
embedded are not only small, they can have extended temperature ranges
of tolerance, use drastically less power and many other features. If
it's purposed hardware, that is only a few things todo, then yes
embedded uP (abbrev for microProcessor) are the way to go. Running off
of 12VDC, means an old car battery and a connection to your solar panels
(assuming you have those) and it's zero on your electric bill. There is
usually a vast array of tax and other incentives, particularly with
solar in Ag businesses.
I've never even seen one of those things, except on my monitor, so I
have no idea what all they are capable of.
Dale, you are pretty strong with Gentoo Linux, so putting a stripped,
purposed, minimized gentoo derivative stack, with far less ebuilds, to
work for your operations, is going to be quite fun. On a farm or ranch,
there are a myriad of things you can do with embedded boards and
gentoo-stripped. You can replace many of those expensive (vendor)
systems with embedded boards +sensors +controls codes and lots of wires
to do most anything. Let's focus on your NAS for now.
I figure a lot of SATA connectors and a ethernet connection
plus enough CPU power and memory to get the job done.
SATA, was great years ago. Still it makes sense to use, if you already
have them. Storage going forward is the process of faster and cheaper
and leaving SATA behind, like ide. Still useful, but a power hog. So
we'll start out with interfacing your existing SATA drives to the
embedded board, and look/decide on options for newer Solid State Data
storage options.
ᅵ https://en.wikipedia.org/wiki/USB
You might not even need many sata ports. usb3 and the upcoming usb4 have
tons of bandwidth (date/time). Mechanical Hard drives are on the way
out. Too expensive and failure prone. SSd and other types of storage,
might be right for you, or a mixture. USB stick memory
can be huge, very low power draw and very inexpensive.
A hybrid of several types of memory storage may be useful to experiment
with. You may want to categorize your long term storage: some accessed
often, others maybe once a year?
For data storage, long term important stuff, you should employ RAID
(1-10). We can get into that later, duplication of important data, via
backups or extra storage is a good idea too. Backups are an old
technology, but may help, but backups do can get old too and fragmented.
For now, lets not worry so much about long term bit integrity, but focus
on your next FUN gentoo rig. I'm hoping other join in to so you have
more than my prospective on your solution.
If those things are capable of doing that fairly
easily.ᅵ After all, I'm me.ᅵ :/
OK, so let's survey some system, you can just purchase
with gentoo preinstalled, or a very easy pathway to embedded gentoo.
Let's look at a few, have some of the other guys jump in, and find you a
solution, to start with. Most will be expandable, and you can figure out
the casing, mounting, power and such.
At this stage, it mostly a research effort and then deciding your
features/price. If you do not have massive bandwitdh requirements, I'm
sure we can find you
a very cost effective, DC powered solution.
Just so you know, I use that fancy $300 OPtima 12vdc charger, and Optima
batteries. the charger reconditions most batteries, if they are not
beyond saving, even cheap lead-acid batteries. Every Farmer should have
one, imho. The Digital 1200 is just awesome.
https://www.optimabatteries.com/en-us/battery-charger
If you like, you can read up on blue, red and yellow top versions and
their intended duty-cycle.
Take some time and look at these gentoo embedded boards, before we
https://wiki.gentoo.org/wiki/Embedded_systems/ARM_hardware_list
https://www.raspberrypi.org/products/raspberry-pi-4-model-b/
Just curious.
Dale
:-)ᅵ :-)
Dale, embedded gentoo is about the most fun thing you'll ever do. A
hardware guy like yourself, that is already an accomplish part of the
great gentoo community, dude,
your going to have a lot of fun, and save money on that farm. So let's
get started!
Post back and I, or others, will try to answer your questions. Be
specific about what you do currently and where you want to end up.
Just so you know, imho, (2) of the finest humans and accomplished Gentoo
devs are listed in gentoo-embedded.
Actually, all the members are just totally awesome humans, so you, my
friend, are in for the ride of a lifetime!
https://wiki.gentoo.org/wiki/Project:Embedded
https://developer.arm.com/tools-and-software/open-source-software/developer-> tools/gnu-toolchain/gnu-rm
More boards to look at (the features).
https://www.embeddedarm.com/products/category/single-board-computers
https://freaklabs.org/chibiarduino/
https://www.reddit.com/r/Gentoo/comments/7oxoln/question_about_gentoo_on_ras
pberry_pi/
Since we're talking about a home NAS and since Dale's always running his PC
24-7, the incremental cost of hanging a few more controllers/drives off it and
using it as a NAS on top of its daily duties, may be the quickest/easiest/most
cost effective approach.
Wols Lists
2020-02-28 11:10:02 UTC
Permalink
Post by james
For data storage, long term important stuff, you should employ RAID
(1-10). We can get into that later, duplication of important data, via
backups or extra storage is a good idea too. Backups are an old
technology, but may help, but backups do can get old too and fragmented.
For now, lets not worry so much about long term bit integrity, but focus
on your next FUN gentoo rig. I'm hoping other join in to so you have
more than my prospective on your solution.
https://raid.wiki.kernel.org/index.php/Linux_Raid

Sorry for the plug, I edit the site. Brickbats/bouquets welcome :-)

Cheers,
Wol
Michael
2020-02-28 11:50:01 UTC
Permalink
Post by Wols Lists
Post by james
For data storage, long term important stuff, you should employ RAID
(1-10). We can get into that later, duplication of important data, via
backups or extra storage is a good idea too. Backups are an old
technology, but may help, but backups do can get old too and fragmented.
For now, lets not worry so much about long term bit integrity, but focus
on your next FUN gentoo rig. I'm hoping other join in to so you have
more than my prospective on your solution.
https://raid.wiki.kernel.org/index.php/Linux_Raid
Sorry for the plug, I edit the site. Brickbats/bouquets welcome :-)
Cheers,
Wol
Thanks for sharing page Wol!

I tried this page from the links at the bottom and ended up in a 404 error:

http://www.runmapglobal.com/blog/fault-tolerant-dedicated-servers/
Wols Lists
2020-02-28 13:30:01 UTC
Permalink
Post by Michael
Post by Wols Lists
Post by james
For data storage, long term important stuff, you should employ RAID
(1-10). We can get into that later, duplication of important data, via
backups or extra storage is a good idea too. Backups are an old
technology, but may help, but backups do can get old too and fragmented.
For now, lets not worry so much about long term bit integrity, but focus
on your next FUN gentoo rig. I'm hoping other join in to so you have
more than my prospective on your solution.
https://raid.wiki.kernel.org/index.php/Linux_Raid
Sorry for the plug, I edit the site. Brickbats/bouquets welcome :-)
Cheers,
Wol
Thanks for sharing page Wol!
http://www.runmapglobal.com/blog/fault-tolerant-dedicated-servers/
Noted. That link pre-dates me working on the site - I haven't checked
all the old links - I guess I should ...

Cheers,
Wol
Peter Humphrey
2020-02-28 16:20:01 UTC
Permalink
--->8
Post by Wols Lists
Post by Michael
http://www.runmapglobal.com/blog/fault-tolerant-dedicated-servers/
Noted. That link pre-dates me working on the site - I haven't checked
all the old links - I guess I should ...
If you run a KDE desktop, KDE link checker is the business.
--
Regards,
Peter.
Wols Lists
2020-02-28 11:10:01 UTC
Permalink
Post by Rich Freeman
A fairly cheap amd64 system can run a ton of services in containers
though, and it is way simpler to maintain that way. I still get quick
access to snapshots/etc, but now if I want to run a gentoo container
it is no big deal if 99% of the time it uses 25MB of RAM and 1% of one
core, but once a month it needs 4GB of RAM and 100% of 6 cores. As
long as I'm not doing an emerge -u world on half a dozen containers at
once it is no big deal at all.
Do all your containers have the same make options etc? Can't remember
which directory it is, but I had a shared emerge directory where it
stored this stuff and I emerged with -bk options (use binary if it's
there, create binary if it isn't).

That way, when I updated my systems, I updated the "big grunt" system
first, then the smaller ones, so the little ones didn't have to emerge
anything other than what was unique to them.

Cheers,
Wol
Rich Freeman
2020-02-28 13:00:02 UTC
Permalink
Post by Wols Lists
Post by Rich Freeman
A fairly cheap amd64 system can run a ton of services in containers
though, and it is way simpler to maintain that way. I still get quick
access to snapshots/etc, but now if I want to run a gentoo container
it is no big deal if 99% of the time it uses 25MB of RAM and 1% of one
core, but once a month it needs 4GB of RAM and 100% of 6 cores. As
long as I'm not doing an emerge -u world on half a dozen containers at
once it is no big deal at all.
Do all your containers have the same make options etc? Can't remember
which directory it is, but I had a shared emerge directory where it
stored this stuff and I emerged with -bk options (use binary if it's
there, create binary if it isn't).
They're probably not too far off in general, but not exact. I only
run one instance of any particular container, so I haven't tried to do
parallel builds. If portage had support for multiple binary packages
co-existing with different build options I might. If I ever get
really bored for a few weeks I could see playing around with that. It
seems like it ought to be possible to content-hash the list of build
options and stick that hash in the binary package filename, and then
have portage search for suitable packages, using a binary package if
one matches, and doing a new build if not.

Many of my containers don't even run Gentoo. I have a few running
Arch, Ubuntu Server, or Debian. If some service is well-supported in
one of those and is poorly supported in Gentoo I will tend to go that
route. I'll package it if reasonable but some upstreams are just not
very conducive to this.

There was a question about ARM-based NAS in this thread which I'll go
ahead and tackle to save a reply. I'm actually playing around with
lizardfs (I might consider moosefs instead if starting from scratch -
or Ceph if I were scaling up but that wouldn't be practical on ARM).
I have a mix of chunkservers but my target is to run new ones on ARM.
I'm using RockPro64 SBCs with LSI HBAs (this SBC is fairly unique in
having PCIe). There is some issue with the lizardfs code that causes
performance issues on ARM though I understand they're working on this,
so that could change. I'm using it for multimedia and I care more
about static space than iops, so it is fine for me. The LSI HBA pulls
more power than the SBC does, but overall the setup is very low-power
and fairly inexpensive (used HBAs on ebay). I can in theory get up to
16 drives on one SBC this way. The SBC also supports USB3 so that is
another option with a hub - in fact I'm mostly shucking USB3 drives
anyway.

Main issue with ARM SBCs in general is that they don't have much RAM,
so IMO that makes Ceph a non-starter. Otherwise that would probably
be my preferred option. Bad things can happen on rebuilds if you
don't have 1GB/TB as they suggest, and even with the relatively
under-utilized servers I have now that would be a LOT of RAM for ARM
(really, it would be expensive even on amd64). Lizardfs/moosefs
chunkservers barely use any RAM at all. The master server does need
more - I have shadow masters running on the SBCs but since I'm using
this for multimedia the metadata server only uses about 100MB of RAM
and that includes processes, libraries, and random minimal service
daemons like sshd. I'm running my master on amd64 though to get
optimal performance, shadowed on the chunkservers so that I can
failover if needed, though in truth the amd64 box with ECC is the
least likely thing to die and runs all the stuff that uses the storage
right now anyway.

The other suggestion to consider USB3 instead of SATA for storage
isn't a bad idea. Though going that route means wall warts and drives
as far as the eye can see. Might still be less messy than my setup,
which has a couple of cheap ATX PSUs with ATX power switches, 16x PCIe
powered risers for the HBAs (they pull too much power for the SBC),
and rockwell drive cages to stack the drives in (they're meant for a
server chasis but they're reasonably priced and basically give you an
open enclosure with a fan). I'd definitely have a lot fewer PCBs
showing if I used USB3 instead. I'm not sure how well that would
perform though - that HBA has a lot of bandwidth if the node got busy
with PCIe v2 x4 connectivity (SAS9200-16E) and with USB3 it would all
go through 1-2 ports. Though I doubt I'd ever get THAT many drives on
a node and if I needed more space I'd probably expand up to 5
chunkservers before I'm putting more than about 3 drives on each - you
get better performance and more fault-tolerance that way.

One big reason I went the distributed filesystem approach was that I
was getting tired of trying to cram as many drives as I could into a
single host and then dealing with some of the inflexibilities of zfs.
The inflexibility bit is improving somewhat with removable vdevs,
though I'm not sure how much residue those leave behind if you do it
often. But, zfs is still limited to however many drives you can cram
into one host, while a distributed filesystem lets you expand
outwards. Plus it is fault-tolerant at the host level instead of the
drive level.
--
Rich
Daniel Frey
2020-02-29 01:20:01 UTC
Permalink
Post by Rich Freeman
Post by james
Yea, I was not clear. I'd run the mail-server, on a 'cluster' (4 or
more), not an individual pi-board unless it was beef up, processor and
ram wise. Gig E would also be on my list.
Unless you have some niche need I wouldn't generally run servers on
Pis. The biggest issue with ARM is that all the cheap platforms are
starved for RAM, and RAM is one of the biggest issues when running
services. And of course the Pi in particular has IO issues (as do
many other cheap SBCs but this is less of an ARM issue). The RAM
issue isn't so many an ARM issue as a supply/demand thing - the only
people asking for 64GB ARM boards are big companies that are willing
to pay a lot for them.
I do actually run a few services on Pis - DNS, DHCP, and a VPN
gateway. That's about it. These are fairly non-demanding tasks that
the hardware doesn't struggle with, and the data is almost entirely
static so an occasional backup makes any kind of recovery trivial.
The only reason I run these services on Pis is that they are fairly
fundamental to having a working network. Most of my services are
running in containers on a server, but I don't want to have to think
about taking a server down for maintenance and then literally every
IOT device in the house won't work. These particular services are
also basically dependency-free which means I can just boot them up and
they just do their jobs, while they remain a dependency for just about
everything else on the network. When you start running DHCP in a
container you have more complex dependency issues.
A fairly cheap amd64 system can run a ton of services in containers
though, and it is way simpler to maintain that way. I still get quick
access to snapshots/etc, but now if I want to run a gentoo container
it is no big deal if 99% of the time it uses 25MB of RAM and 1% of one
core, but once a month it needs 4GB of RAM and 100% of 6 cores. As
long as I'm not doing an emerge -u world on half a dozen containers at
once it is no big deal at all.
Now, if I needed some server in some niche application that needed to
be able to operate off of a car battery for a few days, then sure I'd
be looking at Pis and so on.
Thanks for the detail, I've just ordered an RPi4B to mess around with.
It would be helpful to move DNS etc off my home server as I'm trying to
separate everything into VLANs.

Dan
Rich Freeman
2020-02-29 01:40:01 UTC
Permalink
Post by Daniel Frey
Thanks for the detail, I've just ordered an RPi4B to mess around with.
It would be helpful to move DNS etc off my home server as I'm trying to
separate everything into VLANs.
Keep in mind that Linux supports VLAN tagging, so if you set up your
switch to trunk your server you can have containers or even services
on multiple VLANs on the same host.

I have this configured via systemd-networkd - I'm sure you could do it
with various other network managers as well. I just have a bridge for
each VLAN and then I can attach container virtual ethernet interfaces
to the appropriate VLAN bridge for each container. KVM uses bridges
and it should be just as easy to put VMs on the appropriate bridges.

If you assign IPs on the host to each VLAN interface then as long as
the VLANs don't have conflicting IP addresses you can just attach
services to the appropriate VLANs by binding to their addresses. A
service that binds to 0.0.0.0 or to multiple addresses would listen on
all of them. Now, if your VLANs have conflicting address spaces then
I'd probably just stick to containers so that no host actually sees
conflicting IPs, otherwise you're probably going to have to go crazy
with iproute2 and netfilter to get all the packets going to the right
places.

And all of that should work from a Pi as well as long as long as you
enable CONFIG_VLAN_8021Q. You also need to make sure the tagged VLAN
traffic is passed from the switch (which is not what you normally want
to do for a non-VLAN-aware host where you would filter out all but one
VLAN and remove the tag).

I run my DHCP server on a Pi so that it is more independent.
--
Rich
Daniel Frey
2020-02-29 15:20:01 UTC
Permalink
Post by Rich Freeman
Post by Daniel Frey
Thanks for the detail, I've just ordered an RPi4B to mess around with.
It would be helpful to move DNS etc off my home server as I'm trying to
separate everything into VLANs.
Keep in mind that Linux supports VLAN tagging, so if you set up your
switch to trunk your server you can have containers or even services
on multiple VLANs on the same host.
I have this configured via systemd-networkd - I'm sure you could do it
with various other network managers as well. I just have a bridge for
each VLAN and then I can attach container virtual ethernet interfaces
to the appropriate VLAN bridge for each container. KVM uses bridges
and it should be just as easy to put VMs on the appropriate bridges.
If you assign IPs on the host to each VLAN interface then as long as
the VLANs don't have conflicting IP addresses you can just attach
services to the appropriate VLANs by binding to their addresses. A
service that binds to 0.0.0.0 or to multiple addresses would listen on
all of them. Now, if your VLANs have conflicting address spaces then
I'd probably just stick to containers so that no host actually sees
conflicting IPs, otherwise you're probably going to have to go crazy
with iproute2 and netfilter to get all the packets going to the right
places.
And all of that should work from a Pi as well as long as long as you
enable CONFIG_VLAN_8021Q. You also need to make sure the tagged VLAN
traffic is passed from the switch (which is not what you normally want
to do for a non-VLAN-aware host where you would filter out all but one
VLAN and remove the tag).
I run my DHCP server on a Pi so that it is more independent.
Yes, I'm aware linux does VLANs... I set up netifrc to do this (I
already have some "smart" switches set up - not full layer 3.) I thought
about running containers but if I ever have to do something like
emergency maintenance on my server the whole LAN would be down. Seems
like a no-brainer to have a tiny device like an RPi to do this.

I'm not so sure I'll try installing Gentoo on it though, it doesn't
really seem suitable for compiling tasks. I'm pretty sure the kit I
ordered has a card with Raspbian on it, I'll check that out first.

Dan
Rich Freeman
2020-02-29 15:40:02 UTC
Permalink
Post by Daniel Frey
Yes, I'm aware linux does VLANs... I set up netifrc to do this (I
already have some "smart" switches set up - not full layer 3.) I thought
about running containers but if I ever have to do something like
emergency maintenance on my server the whole LAN would be down. Seems
like a no-brainer to have a tiny device like an RPi to do this.
Yup. It really depends on your requirements.

My main LAN uses a Pi as a DHCP+DNS server, for exactly this reason.
I don't want to be replacing a hard drive in my server and now my
lights/TV/whatever don't work. OpenHab runs on a Pi for this reason
as well.

On the other hand, for my other VLANs DHCP+DNS is handled by stuff
like my UniFi gateway or other embedded solutions. These don't have
the same requirements as my main LAN and being mostly self-contained a
more consumer-oriented solution is fine. I don't want to be doing
security updates on a bazillion Pis either.

I use VLAN on Linux more for providing services on the VLANs. Not
that I have much of this.

Don't think I'm running some kind of datacenter. I just have a
typical home LAN, and I'm running AREDN which basically needs two more
VLANs of its own (one for the network it serves, and one for backhaul
to the internet for tunnels/etc - don't want that stuff getting into
my LAN and the IP address space conflicts in any case). I could see
adding an IOT VLAN maybe, but the problem is that so much of that
stuff needs to interact. If I stuck my TVs/Chromecasts/etc on a
separate VLAN, then I couldn't cast to them from my phone or anything
else unless it was on that VLAN too.
Post by Daniel Frey
I'm not so sure I'll try installing Gentoo on it though, it doesn't
really seem suitable for compiling tasks. I'm pretty sure the kit I
ordered has a card with Raspbian on it, I'll check that out first.
I run Raspbian on my Pis for this reason. If I had some niche use
where Gentoo added value I'd go with it, but otherwise it just seems
too painful.

As it is I have to compile kernel modules on my RockPro64 boards and
that takes forever even without having to build the actual kernel.
When I've built kernels on those while troubleshooting issues with
PCIe it would literally take an hour or more.

If you do want to run Gentoo on a Pi you really should be cross-compiling it.

Something like Gentoo Reference Platform on steroids would certainly
be nice for ARM.
--
Rich
William Kenworthy
2020-03-01 07:20:01 UTC
Permalink
Post by Rich Freeman
Post by Daniel Frey
Yes, I'm aware linux does VLANs... I set up netifrc to do this (I
already have some "smart" switches set up - not full layer 3.) I thought
about running containers but if I ever have to do something like
emergency maintenance on my server the whole LAN would be down. Seems
like a no-brainer to have a tiny device like an RPi to do this.
Yup. It really depends on your requirements.
My main LAN uses a Pi as a DHCP+DNS server, for exactly this reason.
I don't want to be replacing a hard drive in my server and now my
lights/TV/whatever don't work. OpenHab runs on a Pi for this reason
as well.
Keep in mind that rpi are not the only cheap, capable arm hardware out
there.

I am using a number of odroid devices, including an N2 with a gentoo
based kernel and a gentoo aarch64 userland.  Its used for lxc containers
for asterisk, dns, webdav, mail, calendaring and web running on the N2
backed by an Odroid HC2 moosefs cluster (though I am using an intel
powered Odroid H2 for the master).

Its all working rather well now the initial install/config stages are
over.  Part of the gain over the pi's is the use of eMMC storage over
sdcards - almost 5 times faster in my tests.  The 4G of ram has proven
quite adequate so far - even the asterisk latency is better than my
previous QEMU/KVM on intel setup.

I have currently have rpi 1B, 3B and a zero and while the specs for the
rpi4 are better ... its not the best out there.

BillK
Dale
2020-03-01 10:50:01 UTC
Permalink
Post by William Kenworthy
Post by Rich Freeman
Post by Daniel Frey
Yes, I'm aware linux does VLANs... I set up netifrc to do this (I
already have some "smart" switches set up - not full layer 3.) I thought
about running containers but if I ever have to do something like
emergency maintenance on my server the whole LAN would be down. Seems
like a no-brainer to have a tiny device like an RPi to do this.
Yup. It really depends on your requirements.
My main LAN uses a Pi as a DHCP+DNS server, for exactly this reason.
I don't want to be replacing a hard drive in my server and now my
lights/TV/whatever don't work. OpenHab runs on a Pi for this reason
as well.
Keep in mind that rpi are not the only cheap, capable arm hardware out
there.
I am using a number of odroid devices, including an N2 with a gentoo
based kernel and a gentoo aarch64 userland.  Its used for lxc containers
for asterisk, dns, webdav, mail, calendaring and web running on the N2
backed by an Odroid HC2 moosefs cluster (though I am using an intel
powered Odroid H2 for the master).
Its all working rather well now the initial install/config stages are
over.  Part of the gain over the pi's is the use of eMMC storage over
sdcards - almost 5 times faster in my tests.  The 4G of ram has proven
quite adequate so far - even the asterisk latency is better than my
previous QEMU/KVM on intel setup.
I have currently have rpi 1B, 3B and a zero and while the specs for the
rpi4 are better ... its not the best out there.
BillK
Could you share some links to some of these things?  As I mentioned
earlier, I'm thinking about building a NAS system.  Later, I may build a
mythTV system.  Then I can access the NAS from it or my desktop, or cell
phone now that I am somewhat more updated, past the Motorola Razr stage. 

Thanks much. 

Dale

:-)  :-) 
n952162
2020-03-01 11:10:02 UTC
Permalink
...  now that I am somewhat more updated, past the Motorola Razr stage.
Thanks much.
Dale
:-)  :-)
Which Razr do you mean?

Are you 9 years (https://en.wikipedia.org/wiki/Droid_Razr) or 16 years
out of date? (https://en.wikipedia.org/wiki/Motorola_Razr)

;-)

Although, it's amazing to think, today, that there was only 7 years
between those two...
Dale
2020-03-01 13:30:01 UTC
Permalink
Post by n952162
...  now that I am somewhat more updated, past the Motorola Razr stage.
Thanks much.
Dale
:-)  :-)
Which Razr do you mean?
Are you 9 years (https://en.wikipedia.org/wiki/Droid_Razr) or 16 years
out of date? (https://en.wikipedia.org/wiki/Motorola_Razr)
;-)
Although, it's amazing to think, today, that there was only 7 years
between those two...
I had the flip phone, bottom link I think.  I wore out the charging port
and it got to where it was hard to charge the battery.  Otherwise, the
thing still worked great. 

I admit, texting is much easier with the new Samsung.  Also, circle a
word puzzles when at the doctors office waiting helps pass the time too.  :/

Dale

:-)  :-) 
Michael
2020-03-01 11:30:01 UTC
Permalink
Post by William Kenworthy
Post by Rich Freeman
Post by Daniel Frey
Yes, I'm aware linux does VLANs... I set up netifrc to do this (I
already have some "smart" switches set up - not full layer 3.) I thought
about running containers but if I ever have to do something like
emergency maintenance on my server the whole LAN would be down. Seems
like a no-brainer to have a tiny device like an RPi to do this.
Yup. It really depends on your requirements.
My main LAN uses a Pi as a DHCP+DNS server, for exactly this reason.
I don't want to be replacing a hard drive in my server and now my
lights/TV/whatever don't work. OpenHab runs on a Pi for this reason
as well.
Keep in mind that rpi are not the only cheap, capable arm hardware out
there.
I am using a number of odroid devices, including an N2 with a gentoo
based kernel and a gentoo aarch64 userland. Its used for lxc containers
for asterisk, dns, webdav, mail, calendaring and web running on the N2
backed by an Odroid HC2 moosefs cluster (though I am using an intel
powered Odroid H2 for the master).
Its all working rather well now the initial install/config stages are
over. Part of the gain over the pi's is the use of eMMC storage over
sdcards - almost 5 times faster in my tests. The 4G of ram has proven
quite adequate so far - even the asterisk latency is better than my
previous QEMU/KVM on intel setup.
I have currently have rpi 1B, 3B and a zero and while the specs for the
rpi4 are better ... its not the best out there.
BillK
Could you share some links to some of these things? As I mentioned
earlier, I'm thinking about building a NAS system. Later, I may build a
mythTV system. Then I can access the NAS from it or my desktop, or cell
phone now that I am somewhat more updated, past the Motorola Razr stage.
Thanks much.
Dale
:-) :-)
Android has its optimal use cases for sure, but I have found embedded x86 APUs
to be more capable for network and server tasks. As an example:

https://pcengines.ch/
Dale
2020-03-01 14:20:01 UTC
Permalink
Post by Michael
Could you share some links to some of these things? As I mentioned
earlier, I'm thinking about building a NAS system. Later, I may build a
mythTV system. Then I can access the NAS from it or my desktop, or cell
phone now that I am somewhat more updated, past the Motorola Razr stage.
Thanks much.
Dale
:-) :-)
Android has its optimal use cases for sure, but I have found embedded x86 APUs
https://pcengines.ch/
Interesting.  I was thinking at first about just buying a really small
computer MB, like a micro ATX or something, but reading this thread made
think more about something like this.  For one, it pulls a very small
amount of power.  The 'computer' and hard drives would likely pull well
under 50 watts.  Also generate very little heat, more like warmth.  ;-) 

Basically, I'm looking for something that I can build a NAS with.  I'd
need a way to hook 4, 6 maybe 8 hard drives and maybe a couple ethernet
ports.  It seems to be easy to find them with ethernet ports.  It's the
SATA ports that are harder to find.  I found one for the Raspberry thing
that had four ports.  So far, it is the only one I found.  I'm not sure
if it can expand past that either.  It seems the cards are stackable
based on some pics but I'm not sure on that. 

This is interesting info tho.  The link you shared lead me to a tiny
board, maybe a little more powerful than the Raspberry but still low
powered.  Still, the SATA ports is a bit difficult to find.  Maybe I'm
looking for the wrong thing???

Either way, I left a map light, tiny light that's up there with the dome
light, on a few weeks ago in my car.  I pulled out my hydrometer battery
tester.  It's almost always accurate.  I got a few week cells in my
car's battery.  After someone else posted about those AGM batteries, I'm
thinking of going down that route even for my car.  They ain't cheap by
no stretch.  That said, those Raspberry things are almost disposable if
you get a Chinese model.  lol 

Dale

:-)  :-) 
Rich Freeman
2020-03-01 14:40:02 UTC
Permalink
Post by William Kenworthy
Keep in mind that rpi are not the only cheap, capable arm hardware out
there.
I completely agree. Anytime I'm looking at an application I consider
the SBCs available as options. Certainly the odroids are highly
spoken of.

Main advantage of the Pi is its ubiquity - just about anything you
could want is already packaged and documented for it. It is also
pretty cheap.
Post by William Kenworthy
backed by an Odroid HC2 moosefs cluster (though I am using an intel
powered Odroid H2 for the master).
I considered an HC2 for lizardfs. My problem with it is that it has a
single SATA port, which means you're buying a $50 SBC for every hard
drive in your cluster.

For a single drive per node it is probably your best bet. However, my
chunkservers are:
~$65 RockPro64
$20 used LSI HBA
$5 wall wart
$25 cheap ATX PSU
$5 ATX power switch
$5 extra SATA cables
$5 powered 16x PCIe riser cable (these are a bit hard to find)

That is ~$125, and will support 16 hard drives. You're saving money
on the 3rd drive per node. If you want some kind of enclosure for the
drives you'll pay maybe another $5/drive.

The other option that might be worth considering if you don't mind
losing some bandwidth to the drives is just using SATA3 and hubs/etc
and external drives. I'm shucking external drives anyway. So, any
SBC with a SATA3 port would work for that, with nothing else needed.
I could see USB3 bandwidth (shared) being a constraint if you're
rebuilding, but it would keep up with gigabit ethernet.

Oh, and for any kind of NAS/etc solution make sure that whatever you
get has gigabit ethernet. The Pi3s at least don't have that - not
sure about the Pi4. Wouldn't help in a Pi3 anyway as I think the LAN
goes through the internal USB2 bus - the Pi is pretty lousy for IO in
general - at least conventional PC IO. That GPIO breakout is of
course nice for projects.
--
Rich
Daniel Frey
2020-03-01 15:40:01 UTC
Permalink
Post by Rich Freeman
Post by William Kenworthy
Keep in mind that rpi are not the only cheap, capable arm hardware out
there.
I completely agree. Anytime I'm looking at an application I consider
the SBCs available as options. Certainly the odroids are highly
spoken of.
Main advantage of the Pi is its ubiquity - just about anything you
could want is already packaged and documented for it. It is also
pretty cheap.
Post by William Kenworthy
backed by an Odroid HC2 moosefs cluster (though I am using an intel
powered Odroid H2 for the master).
I considered an HC2 for lizardfs. My problem with it is that it has a
single SATA port, which means you're buying a $50 SBC for every hard
drive in your cluster.
For a single drive per node it is probably your best bet. However, my
~$65 RockPro64
$20 used LSI HBA
$5 wall wart
$25 cheap ATX PSU
$5 ATX power switch
$5 extra SATA cables
$5 powered 16x PCIe riser cable (these are a bit hard to find)
That is ~$125, and will support 16 hard drives. You're saving money
on the 3rd drive per node. If you want some kind of enclosure for the
drives you'll pay maybe another $5/drive.
The other option that might be worth considering if you don't mind
losing some bandwidth to the drives is just using SATA3 and hubs/etc
and external drives. I'm shucking external drives anyway. So, any
SBC with a SATA3 port would work for that, with nothing else needed.
I could see USB3 bandwidth (shared) being a constraint if you're
rebuilding, but it would keep up with gigabit ethernet.
Oh, and for any kind of NAS/etc solution make sure that whatever you
get has gigabit ethernet. The Pi3s at least don't have that - not
sure about the Pi4. Wouldn't help in a Pi3 anyway as I think the LAN
goes through the internal USB2 bus - the Pi is pretty lousy for IO in
general - at least conventional PC IO. That GPIO breakout is of
course nice for projects.
I was reading the Pi4 has true gigabit now, thanks to its USB3 ports.

Dan
William Kenworthy
2020-03-02 02:00:01 UTC
Permalink
Post by William Kenworthy
Keep in mind that rpi are not the only cheap, capable arm hardware out
there.
I am in Oz, delivery from HardKernel (the South Korean company behind
the Odroid line) takes ~1 week.  Shipping is mostly via FedEx, who are a
bit pricy and for me means a 30m drive to get it from a delivery centre
as I cant be home to take delivery - though the last delivery used a
different, much more flexible delivery company.

lizardfs and moosefs are very similar (originally a fork) - I went with
moosefs as the community was better a few months ago - but lizardfs
sounds like its getting back on track after some infighting (moosefs
apparently went through the same thing)  I find the moosefs
documentation and community help are adequate, but ultimately they are
are a commercial project with a free taster offering hence some
limitations in design for the community version (such as single
master/no shadow masters) that lizardfs doesn't have.  I will move to
lizardfs when I am satisfied they have their act together because of
this - having a single master means taking the whole cluster offline
when the master needs maintenance or fails which is painful

For those wanting to run a lot of drives on a single host - that defeats
the main advantage of using a chunkserver based filesystem -
redundancy.  Its far more common to have a host fail than a disk drive. 
Losing the major part of your storage in one go means the cluster is
effectively dead - hence having a lot of completely separate systems is
much more reliable - yes, I did try having 4 sata drives on an atom
board and found it was easy to justify two more HC2's for the
reliability. (note, its not the effect of the atom boards reliability I
am pointing out, but the effect on the whole storage system of losing
such a large percentage of capacity in one go - think maintenance, fail
to startup etc. - its more common than you may think)

Dale: there is no single reference - I just designed on the fly and did
what was necessary to move from my existing KVM/Qemu architecture on a
two powerfull intel systems, each with 8TB storage to an lxc container
based system backed by moosefs.  It uses an odroid arm based n2 for lxc,
an arm based xu4 for management and software compilation (its not that
powerfull, but is adequate), the HC2's (very similar to the xu4 - same
arm architecture) and an H2 to run the mfsmaster and I am moving to
ansible for management.  Overall, I am finding less maintenance due to
better backups and better reliability, considerably lower power
consumption while overall performance seems better.

Gotchas:

1. I originally got the 4gb ram N2 to run the mfsmaster software - tests
were excellent - until I added multiple copies of a mailserver with
nearly 20 years of history - hit swap and slowed to a crawl.  So I now
have an intel based Odroid H2 with 32GB ram - currently is using ~4.2GB
ram for 7TB of 20TB used, but has hit over 32GbB and well into swap
while converging.  One admin in our local LUG is using 4xHC2 with the
master running on one of the HC2's as a media server with no problems -
its millions of small files added ina short time that causes the grief -
there is a formula in the moosefs documentation allowing resources to be
estimated - on the mailing list I saw a mention of a data centre using
something like 150G ram and having problems!

2. The Odroid HC2 has a single sata port and a single usb port - I have
5 of them, two have a sata + a usb3 drive attached.  I did try (on the
N2) using multiple usb3 drives on the internal hub - disaster with way
too much traffic through the hub - don't do it!

3. Storage options are an SD card (the faster the better - swap on an SD
card ... sucks!) or eMMC (5xfaster than even a good SD card.) for the N2
and xu4 - HC2's can only do sdcard, or sata. The H2 can do sd card,
sata, eMMC, or m2 NVME - this last really flies!  Note that almost all
arm system max out at 2 to 4GB of ram, so swap is usually needed for
safety - depending on OOM killer resource management on a SAN type
storage system is asking for corruption like I saw in one recent gentoo
email.

4. xu4/HC2 are 32 bit arm v7, the N2 is 64 bit and runs aarch64 nicely -
I copied my rpi images across repurposed them (hooray for
emerge/portage!).  I do not use, or have done any work on their
graphics/multimedia capabilities

5. I want to move to all gentoo-sources kernels - the xu4/HC2's are
still on the odroid kernel until I get around to it.  The N2 was a lot
of work, but ultimately successful, the H2 is standard amd64 EFI.


Have fun!

BillK
Rich Freeman
2020-03-02 02:50:01 UTC
Permalink
Post by William Kenworthy
For those wanting to run a lot of drives on a single host - that defeats
the main advantage of using a chunkserver based filesystem -
redundancy. Its far more common to have a host fail than a disk drive.
Losing the major part of your storage in one go means the cluster is
effectively dead - hence having a lot of completely separate systems is
much more reliable
Of course. You should have multiple hosts before you start putting
multiple drives on a single host.

However, once you have a few hosts the performance improves by adding
more, but you're not really getting THAT much additional redundancy.
You would get faster rebuild times by having more hosts since there
would be less data to transfer when one fails and more hosts doing the
work.

So, it is about finding a balance. You probably don't want 30 drives
on 2 hosts. However, you probably also don't need 15-30 hosts for
that many drives either. I wouldn't be putting 16 drives onto a
single host until I had a fair number of hosts.

As far as the status of lizardfs goes - as far as I can tell it is
mostly developed by a company and they've wavered a bit on support in
the last year. I share your observation that they seem to be picking
up again. In any case, I'm running the latest stable and it works
just fine, but it lacks the high availability features. I can have
shadow masters, but they won't automatically fail over, so maintenance
on the master is still a pain. Recovery due to failure of the master
should be pretty quick though even if manual - just have to run a
command on each shadow to determine which has the most recent
metadata, then adjust DNS for my master CNAME to point to the new
master, and then edit config on the new master to tell it that it is
the master and no longer a shadow, and after restarting the daemon the
cluster should be online again.

The latest release candidate has the high availability features (used
to be paid, is now free), however it is still a release candidate and
I'm not in that much of a rush. There was a lot of griping on the
forums/etc by users who switched to the release candidate and ran into
bugs that ate their data. IMO that is why you don't go running
release candidates for distributed filesystems with a dozen hard
drives on them - if you want to try them out just run them in VMs with
a few GB of storage to play with and who cares if your test data is
destroyed. It is usually wise to be conservative with your
filesystems. Makes no difference to me if they take another year to
do the next release - I'd like the HA features but it isn't like the
old code goes stale.

Actually, the one thing that it would be nice if they fixed is the
FUSE client - it seems to leak RAM.

Oh, and the docs seem to hint at a windows client somewhere which
would be really nice to have, but I can't find any trace of it. I
only normally run a single client but it would obviously perform well
as a general-purpose fileserver.

There has been talk of a substantial rewrite, though I'm not sure if
that will actually happen now. If it does I hope they do keep the RAM
requirements low on the chunkservers. That was the main thing that
turned me off from ceph - it is a great platform in general but
needing 1GB RAM per 1TB disk adds up really fast, and it basically
precludes ARM SBCs as OSDs as you can't get those with that much RAM
for any sane price - even if you were only running one drive per host
good luck finding a SBC with 13GB+ of RAM. You can tune ceph to use
less RAM but I've heard that bad things happen if you have some hosts
shuffle during a rebuild and you don't have gobs of RAM - all the OSDs
end up with an impossible backlog and they keep crashing until you run
around like Santa Claus filling every stocking with a handful of $60
DIMMs.

Right now lizardfs basically uses almost no ram at all on
chunkservers, so an ARM SBC could run dozens of drives without an
issue.
--
Rich
William Kenworthy
2020-03-02 05:30:01 UTC
Permalink
Post by Rich Freeman
Post by William Kenworthy
For those wanting to run a lot of drives on a single host - that defeats
the main advantage of using a chunkserver based filesystem -
redundancy. Its far more common to have a host fail than a disk drive.
Losing the major part of your storage in one go means the cluster is
effectively dead - hence having a lot of completely separate systems is
much more reliable
Of course. You should have multiple hosts before you start putting
multiple drives on a single host.
However, once you have a few hosts the performance improves by adding
more, but you're not really getting THAT much additional redundancy.
You would get faster rebuild times by having more hosts since there
would be less data to transfer when one fails and more hosts doing the
work.
So, it is about finding a balance. You probably don't want 30 drives
on 2 hosts. However, you probably also don't need 15-30 hosts for
that many drives either. I wouldn't be putting 16 drives onto a
single host until I had a fair number of hosts.
As far as the status of lizardfs goes - as far as I can tell it is
mostly developed by a company and they've wavered a bit on support in
the last year. I share your observation that they seem to be picking
up again. In any case, I'm running the latest stable and it works
just fine, but it lacks the high availability features. I can have
shadow masters, but they won't automatically fail over, so maintenance
on the master is still a pain. Recovery due to failure of the master
should be pretty quick though even if manual - just have to run a
command on each shadow to determine which has the most recent
metadata, then adjust DNS for my master CNAME to point to the new
master, and then edit config on the new master to tell it that it is
the master and no longer a shadow, and after restarting the daemon the
cluster should be online again.
The latest release candidate has the high availability features (used
to be paid, is now free), however it is still a release candidate and
I'm not in that much of a rush. There was a lot of griping on the
forums/etc by users who switched to the release candidate and ran into
bugs that ate their data. IMO that is why you don't go running
release candidates for distributed filesystems with a dozen hard
drives on them - if you want to try them out just run them in VMs with
a few GB of storage to play with and who cares if your test data is
destroyed. It is usually wise to be conservative with your
filesystems. Makes no difference to me if they take another year to
do the next release - I'd like the HA features but it isn't like the
old code goes stale.
Actually, the one thing that it would be nice if they fixed is the
FUSE client - it seems to leak RAM.
Oh, and the docs seem to hint at a windows client somewhere which
would be really nice to have, but I can't find any trace of it. I
only normally run a single client but it would obviously perform well
as a general-purpose fileserver.
There has been talk of a substantial rewrite, though I'm not sure if
that will actually happen now. If it does I hope they do keep the RAM
requirements low on the chunkservers. That was the main thing that
turned me off from ceph - it is a great platform in general but
needing 1GB RAM per 1TB disk adds up really fast, and it basically
precludes ARM SBCs as OSDs as you can't get those with that much RAM
for any sane price - even if you were only running one drive per host
good luck finding a SBC with 13GB+ of RAM. You can tune ceph to use
less RAM but I've heard that bad things happen if you have some hosts
shuffle during a rebuild and you don't have gobs of RAM - all the OSDs
end up with an impossible backlog and they keep crashing until you run
around like Santa Claus filling every stocking with a handful of $60
DIMMs.
Right now lizardfs basically uses almost no ram at all on
chunkservers, so an ARM SBC could run dozens of drives without an
issue.
Everything bad you hear about ceph is true ... and then some! I did try,
but this was some years ago so hopefully its better now. The two biggies
were excessive network requirements (bandwidth, separation) and recovery
times with frequent crash and burn. There are ceph features I would
really like to use (rbd, local copies with much simpler config, ...) but
moosefs is a lot more bullet proof on lesser resource requirements
though I did find properly pruned vlans on a smartswitch separating the
intra-cluster from external requests made a noticeable difference.

moosefs has a windows client but its only available with the paid
version.  The master/shadow-master and auto failover is only available
through the paid version - for the community you have to stop the
master, copy the files then change DNS etc. before restarting the new
master - cant really do it online even when scripted - its painful with
downtime and I had dns caching issues that took time to work their way
out of the system.  I thought lizardfs was much more community minded
but you are characterising it as similar to moosefs - a taster offering
by a commercial company holding back some of the non-essential but
jucier features for the paid version - is that how you see them?

By the way, to keep to the rpi subject, I did have a rpi3B with a usb2
sata drive attached but it was hopeless as a chunkserver impacting the
whole cluster.  Having the usb data flow and network data flow through
the same hub just didn't go well - I started with the odroids before the
rpi4 was released or I might have experimented with it first (using a
sata HAT) - anyone with a comment on how that compares with a HC2?

BillK
Rich Freeman
2020-03-02 14:50:01 UTC
Permalink
Post by William Kenworthy
I thought lizardfs was much more community minded
but you are characterising it as similar to moosefs - a taster offering
by a commercial company holding back some of the non-essential but
jucier features for the paid version - is that how you see them?
I don't see much of an active community. It seems like most actual
development happens outside of the public repo, with big code drops by
the private team doing the work (which seems to be associated with a
company). A bit like the Android model. I'm sure they'll accept pull
requests, but that isn't how most of the work is getting done.

It seems like the main difference between them and moosefs is that
they're making more stuff FOSS to entice users over. Shadow masters
are FOSS as opposed to just having metadata loggers. HA is FOSS in
the latest RC.

So, it seems like their model is to trickle out the non-free stuff and
make it free after a delay.

It really seems like Ceph is the best fully open platform out there,
but the resource requirements just make it impractical. I have no
doubt that it can scale FAR better with its design, but that design
basically forces every node to be a bit of a powerhouse, versus
Lizardfs where you just have one daemon with all the intelligence and
the rest are just dumping files on disks. And you really don't need
much CPU/RAM for the master if you're serving large files - the
demands would go up with IOPS and number of files, and multimedia is
low on both.
Post by William Kenworthy
By the way, to keep to the rpi subject, I did have a rpi3B with a usb2
sata drive attached but it was hopeless as a chunkserver impacting the
whole cluster. Having the usb data flow and network data flow through
the same hub just didn't go well
Hard drives plus 100Mbps LAN sharing a single USB 2.0 hub is
definitely not a recipe for NAS success...

When I upgraded to UniFi switches I really only noticed for the first
time how many hosts I have that aren't gigabit, and they're mostly Pis
at this point. They're nice little project boards but for anything
IO-intensive they're almost always the wrong choice.

The RockPro64 I'm using has gigabit plus PCIe 3.0 x8 plus USB3 and as
far as I can tell they don't have any contention. Maybe they're all
on a PCIe bus or something but obviously that can handle quite a bit.
Only issue was that the rk3399 PCIe drivers were not the most robust
in the kernel, but ayufan and the IRC channel were both helpful and
his kernel branch is actively maintained, so I was able to get
everything sorted (some delays needed during training to allow boards
to initialize and I was having power issues in the beginning). Much
of the rk3399 support in the kernel was pushed by Google for
Chromebooks and LSI HBAs weren't exactly on their list of things to
test with those - not sure if the Chromebooks put much of anything on
PCIe.
--
Rich
Daniel Frey
2020-03-01 15:40:02 UTC
Permalink
Post by William Kenworthy
Keep in mind that rpi are not the only cheap, capable arm hardware out
there.
I am using a number of odroid devices, including an N2 with a gentoo
based kernel and a gentoo aarch64 userland.  Its used for lxc containers
for asterisk, dns, webdav, mail, calendaring and web running on the N2
backed by an Odroid HC2 moosefs cluster (though I am using an intel
powered Odroid H2 for the master).
Its all working rather well now the initial install/config stages are
over.  Part of the gain over the pi's is the use of eMMC storage over
sdcards - almost 5 times faster in my tests.  The 4G of ram has proven
quite adequate so far - even the asterisk latency is better than my
previous QEMU/KVM on intel setup.
I have currently have rpi 1B, 3B and a zero and while the specs for the
rpi4 are better ... its not the best out there.
BillK
I am aware of other devices but the RPi (afaict) is the only one sold
within the country and not subject to import duty/fees/taxes and the
related shipping delays because of those. :(

Dan
n952162
2020-03-01 15:50:02 UTC
Permalink
"within the country"?  :-)  You must be American?
Post by Daniel Frey
Post by William Kenworthy
Keep in mind that rpi are not the only cheap, capable arm hardware out
there.
I am using a number of odroid devices, including an N2 with a gentoo
based kernel and a gentoo aarch64 userland.  Its used for lxc containers
for asterisk, dns, webdav, mail, calendaring and web running on the N2
backed by an Odroid HC2 moosefs cluster (though I am using an intel
powered Odroid H2 for the master).
Its all working rather well now the initial install/config stages are
over.  Part of the gain over the pi's is the use of eMMC storage over
sdcards - almost 5 times faster in my tests.  The 4G of ram has proven
quite adequate so far - even the asterisk latency is better than my
previous QEMU/KVM on intel setup.
I have currently have rpi 1B, 3B and a zero and while the specs for the
rpi4 are better ... its not the best out there.
BillK
I am aware of other devices but the RPi (afaict) is the only one sold
within the country and not subject to import duty/fees/taxes and the
related shipping delays because of those. :(
Dan
Daniel Frey
2020-03-02 00:10:01 UTC
Permalink
Post by n952162
"within the country"?  :-)  You must be American?
No.

Dan
n952162
2020-03-02 07:40:02 UTC
Permalink
Sorry for the presumption.
Post by n952162
"within the country"?  :-)  You must be American?
No.
Dan
Loading...