Post by IanDPost by Arne VajhøjPost by IanDPost by Arne VajhøjPost by IanDIn my place, I'd desperately love to investigate RabbitMQ as I
believe it could be of use but when I look at the Erlang port
and read comments it's as though everything fell in a hole and
was not just hidden but was buried :-(
Any reason why you particular want RabbitMQ.
I am asking because the roadmap that started this thread claims
that ActiveMQ is available.
Since I'm starting out learning everything I have come across has
said Rabbit is the most popular
Here is one such source, there's plenty of others
https://stackshare.io/stackups/rabbitmq-vs-activemq-vs-zeromq
It is possible that RabbitMQ is more widely used than ActiveMQ.
But I will say that ActiveMQ is so widely used that it is an
acceptable option.
Post by IanDIt seems in my preliminary looking around, Rabbit has more frequently
updated python support as well
Again - very possible.
But ActiveMQ does have Python support.
Post by IanDWhether true or not, I've run across people saying they have found
Rabbit to be more stable which they attribute to Erlang's OTP design
principals
ActiveMQ is pretty stable.
Overall I think you should give ActiveMQ a try. Its status
in VSI roadmap may make up for other aspects.
Arne
Thanks
And for more background details...
At this stage I am just looking at possible ideas as I started to look at RabbitMQ
Basically, we have an application that takes events from different hardware and they get bundled on some upstream system(s) that then collate them and ftp them onto the VMS system as bundled event file(s)
There are approximately 200K files in an 8 hour period, ranging in size from a few k to about 5M on the upper size of things
The number of actual events would be about 2 million (but they are bundled into files that are ftp's to us)
So at around peak, we would be receiving via ftp about 10 - 12 files/sec.
Nothing really large but large enough as the information basically needs to be unpacked (from the ftp file and processed).
Once the processing is done, they are bundled again into files with additional information and ftp'd to other downstream systems for customer consumption (display only)
There is talk about a 400% increase in events so I naturally started to wonder if the current ftp method was really up to the task and therefore started to wonder about message queuing as an alternative - in particular the idea of message queuing having the ability to broadcast which may allow some parallel processing work to be done on the flowing data versus a strict waterfall approach (this is just an idea, i have not looked into the feasibility exactly)
In a previous role, we used a webserver (WASD) that handled this type of task well
We do not have a relational DB and everything is flat file based with near zero chance of moving to a relational DB. They would rather spend the money on decommissioning than go to the expense of moving to a relational DB (in fact, this is the long term plan - story of my life in regards to VMS it seems - everyone wanting to decommission...)
Well Ian, you give scant details. So I would ask some questions.
Is there any reason to bundle the data? Sounds like you must then unbundle it
at some point.
If the sender(s) and receiver(s) are not always active, then an async message
system could be a solution. Volume would be a consideration there.
However, if they are always available, and that would mean that if one was down
sync communications would also be down. If you can do sync communications, then
perhaps a sync communications system might be appropriate. Listener sockets and
clients can do that. You can also keep up a permanent link.
More versatile would be async communications. Using automated messages would
allow better automation of the task(s). You could process each individual
packet of data as it arrived.
Note that such a procedure isn't something for a system manager to implement.
Software architects and engineers seem to be called for. That of course costs
money, and you have indicated in the past that even if you could offer these
people the best system in the world, they still would not want it.
Note, a RDBMS would not necessarily be of more benefit in what you describe.
Text files and such can store data just as well as anything else. It's in
retrieval that a RDBMS shines. For temporary data, the flat files are probably
a better solution. Actually, with messages, no files are required. "A" sends
data to "B" in a message, "B" processes the data, then sends data to "C" in
another message. It may be that at some point some data may be stored for
historical purposes. A RDBMS would be good for that. But there are other methods.