Please give 2.0.6 a try when it comes out in a week or two. If you
want to test now you can use the dev/unstable 2.1.1 release.
On Thu, May 10, 2012 at 11:13 AM, Shi Shei <QTRAURFUICIN-geNee64TY+gS+***@public.gmane.org> wrote:
> Put me in the list as well. We are suffering from exactly the same issues
> that Prasanna is describing.
> Mongo pretends to be fast with commodity hardware when you use sharding but
> even with sharding and high performance hardware we don't see it.
>
> Have a look at our MMS (group name "idealo", all hosts containing
> "offerstore"). Profiling is enabled for queries taking longer than 3 seconds
> - and there are a lot of them!
> We followed and have mostly applied 10gens production rules but the results
> are still invisible.
> Sorry to say but with such expensive hardware, mongo performance is not that
> good as expected.
>
> If you have some pointers, we'll gladly follow your instructions. However,
> just waiting for the next and still next mongo release is quite
> demotivating.
>
> On Thursday, May 10, 2012 5:04:33 PM UTC+2, Scott Hernandez wrote:
>>
>> You seem to looking at this as a binary result. Slow reads do not mean
>> downtime. There are many use-cases where many times performance
>> degradation during maintenance windows is acceptable but downtime
>> (stopping reads) is not.
>>
>> We are working with everyone to make sure they can get the best
>> performance and uptime for their systems. If you have more specific
>> areas where you are seeing issue, please create a new thread/post or
>> jira issue with the problem (or vote/watch for existing ones).
>>
>> Nobody thinks the system is perfect as is (how could it be when there
>> are so many different use cases), and there are lots of improvements,
>> fixes and changes going in to address these areas for improvement.
>>
>>
>> On Thu, May 10, 2012 at 7:44 AM, Prasanna .S <myprasanna-***@public.gmane.org> wrote:
>> > That use case can easily be supported by just shutting off the reads
>> > from
>> > the application. Basically they are anyways taking downtime -- that
>> > doesn't
>> > take special support, does it?
>> >
>> > In the normal case, don't you think the reads are the highest priority?
>> > Come
>> > on, this is production.. And I'm telling you, your top clients have this
>> > problem. Whats worse for a DB than downtime?
>> >
>> >
>> > On Thu, May 10, 2012 at 7:37 AM, Scott Hernandez
>> > <scotthernandez-***@public.gmane.org>
>> > wrote:
>> >>
>> >> A simple example is a system where they want the balancer to run as
>> >> quickly as possible at low traffic times to balancer data and don't
>> >> care about the impact of (user) reads. This can be a big deal when you
>> >> are already under-provisioned and need to add a new shard to handle
>> >> users more effectively -- it has come up a few times.
>> >>
>> >> On Thu, May 10, 2012 at 7:33 AM, Prasanna .S <myprasanna-***@public.gmane.org>
>> >> wrote:
>> >> > Scott,
>> >> >
>> >> >> While it might be paramount for some situations/use-cases to
>> >> >> complete all
>> >> >> reads before a move starts, it might not be for others (which I
>> >> > have seen).
>> >> >
>> >> > Can you please explain the "might not be for others" part? Can you
>> >> > give
>> >> > me
>> >> > an example?
>> >> >
>> >> >
>> >> > On Thu, May 10, 2012 at 7:19 AM, Scott Hernandez
>> >> > <scotthernandez-***@public.gmane.org>
>> >> > wrote:
>> >> >>
>> >> >> There are lots of ways to look at this with some competing goals.
>> >> >> While it might be paramount for some situations/use-cases to
>> >> >> complete
>> >> >> all reads before a move starts, it might not be for others (which I
>> >> >> have seen). This fix will allow reads (and other writes) to get
>> >> >> processes while chunks moves happen, period. The issue of locking,
>> >> >> and
>> >> >> contentions on those locks, is very important and there are lots of
>> >> >> improvements being made which help across the boards.
>> >> >>
>> >> >> Feel free to add comments or create new issues with your ideas of
>> >> >> how
>> >> >> your system would like things to best behave:
>> >> >> http://jira.mongodb.org/browse/server
>> >> >>
>> >> >> On Wed, May 9, 2012 at 7:46 PM, Prasanna .S <myprasanna-***@public.gmane.org>
>> >> >> wrote:
>> >> >> > Scott,
>> >> >> >
>> >> >> > The below are heuristics, and not guarantees. What you want from a
>> >> >> > DB
>> >> >> > stand-point is the latter.
>> >> >> >
>> >> >> > The delay between moves, won't provide a guarantee, unless you
>> >> >> > make
>> >> >> > sure
>> >> >> > that
>> >> >> > (1) the next iteration of moveChunk, does not start unless all
>> >> >> > pending
>> >> >> > reads
>> >> >> > complete.
>> >> >> > (2) Each iteration takes less than 1second.
>> >> >> >
>> >> >> > Thoughts?
>> >> >> >
>> >> >> >
>> >> >> > On Wed, May 9, 2012 at 6:18 PM, Scott Hernandez
>> >> >> > <scotthernandez-***@public.gmane.org>
>> >> >> > wrote:
>> >> >> >>
>> >> >> >> The logic there is to do each document write, during the migrate,
>> >> >> >> in
>> >> >> >> a
>> >> >> >> separate lock so other operations can be interleaved. We have
>> >> >> >> seen
>> >> >> >> this change dramatically reduce the constant lock percentage and
>> >> >> >> allow
>> >> >> >> reads, and other writes through during a initial move.
>> >> >> >>
>> >> >> >> There is already a delay between moves which doesn't seem to be
>> >> >> >> your
>> >> >> >> problem.
>> >> >> >>
>> >> >> >> On Wed, May 9, 2012 at 5:37 PM, Prasanna .S
>> >> >> >> <myprasanna-***@public.gmane.org>
>> >> >> >> wrote:
>> >> >> >> > Scott,
>> >> >> >> >
>> >> >> >> > I tried to understand what's going on there. But with a quick
>> >> >> >> > glance,
>> >> >> >> > I
>> >> >> >> > assume, a lock was being held longer than necessary and was
>> >> >> >> > fixed.
>> >> >> >> > It's
>> >> >> >> > unclear how this can help me. The only way to reliably make
>> >> >> >> > sure
>> >> >> >> > the
>> >> >> >> > site
>> >> >> >> > keeps running while a moveChunk is executing, is if, you yield
>> >> >> >> > every
>> >> >> >> > second
>> >> >> >> > within moveChunk and make sure all pending reads complete,
>> >> >> >> > before
>> >> >> >> > the
>> >> >> >> > next
>> >> >> >> > moveChunk period starts.
>> >> >> >> >
>> >> >> >> > Are you guys doing it, or planning to do it? Is there any other
>> >> >> >> > high-level
>> >> >> >> > way to guarantee the reads can be answered within a couple of
>> >> >> >> > seconds?
>> >> >> >> >
>> >> >> >> > Thanks
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > On Wed, May 9, 2012 at 5:17 PM, Scott Hernandez
>> >> >> >> > <scotthernandez-***@public.gmane.org>
>> >> >> >> > wrote:
>> >> >> >> >>
>> >> >> >> >> Okay, looking at the receiving shard mongo55, and donor shard
>> >> >> >> >> mongo43,
>> >> >> >> >> it look like the fix in 2.0.6 (due out in the next two weeks
>> >> >> >> >> of
>> >> >> >> >> so)
>> >> >> >> >> will help you out a lot.
>> >> >> >> >>
>> >> >> >> >> On Wed, May 9, 2012 at 4:37 PM, Prasanna .S
>> >> >> >> >> <myprasanna-***@public.gmane.org>
>> >> >> >> >> wrote:
>> >> >> >> >> > 4:37 the balancer has started and disk shot up.
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > On Wed, May 9, 2012 at 4:30 PM, Prasanna .S
>> >> >> >> >> > <myprasanna-***@public.gmane.org>
>> >> >> >> >> > wrote:
>> >> >> >> >> >>
>> >> >> >> >> >> Hey Scott,
>> >> >> >> >> >>
>> >> >> >> >> >> I'm going to do this now. Will start it and stop it quickly
>> >> >> >> >> >> and
>> >> >> >> >> >> give
>> >> >> >> >> >> you
>> >> >> >> >> >> the exact times.
>> >> >> >> >> >>
>> >> >> >> >> >> Thanks,
>> >> >> >> >> >> -Prasanna
>> >> >> >> >> >>
>> >> >> >> >> >>
>> >> >> >> >> >> On Wed, May 9, 2012 at 3:41 PM, Scott Hernandez
>> >> >> >> >> >> <scotthernandez-***@public.gmane.org>
>> >> >> >> >> >> wrote:
>> >> >> >> >> >>>
>> >> >> >> >> >>> Those values are in blocks (512 bytes), not MB.
>> >> >> >> >> >>>
>> >> >> >> >> >>> We have an update in the upcoming 2.0.6 stable release
>> >> >> >> >> >>> which
>> >> >> >> >> >>> will
>> >> >> >> >> >>> help
>> >> >> >> >> >>> with migrations a bit.
>> >> >> >> >> >>> https://jira.mongodb.org/browse/SERVER-5303
>> >> >> >> >> >>> You
>> >> >> >> >> >>> can click on the version like to see other important
>> >> >> >> >> >>> fixes.
>> >> >> >> >> >>>
>> >> >> >> >> >>> Can you do a few migrates tonight so we can see the
>> >> >> >> >> >>> stats/behavior
>> >> >> >> >> >>> in
>> >> >> >> >> >>> MMS? Please let us know when you do this so we can take a
>> >> >> >> >> >>> look
>> >> >> >> >> >>> at
>> >> >> >> >> >>> the
>> >> >> >> >> >>> data to see if this fix will help.
>> >> >> >> >> >>>
>> >> >> >> >> >>>
>> >> >> >> >> >>> On Tue, May 8, 2012 at 11:49 AM, Prasanna .S
>> >> >> >> >> >>> <myprasanna-***@public.gmane.org>
>> >> >> >> >> >>> wrote:
>> >> >> >> >> >>> > Hey Dan,
>> >> >> >> >> >>> >
>> >> >> >> >> >>> > Read ahead settings are the defaults (256
>> >> >> >> >> >>> > MB): http://pastebin.com/MHc8s2fq
>> >> >> >> >> >>> > Let me know if you need something else.
>> >> >> >> >> >>> >
>> >> >> >> >> >>> > @Elliot,
>> >> >> >> >> >>> >
>> >> >> >> >> >>> > I spoke with some of my friends at foursquare and it
>> >> >> >> >> >>> > turns
>> >> >> >> >> >>> > out
>> >> >> >> >> >>> > they
>> >> >> >> >> >>> > face the
>> >> >> >> >> >>> > same issue as well. I've had my other friends who run
>> >> >> >> >> >>> > companies >
>> >> >> >> >> >>> > $100M
>> >> >> >> >> >>> > market cap (whom I'd rather not name in this list), tell
>> >> >> >> >> >>> > me
>> >> >> >> >> >>> > about
>> >> >> >> >> >>> > their
>> >> >> >> >> >>> > mongo problems in the past few days. This is pretty
>> >> >> >> >> >>> > serious
>> >> >> >> >> >>> > stuff
>> >> >> >> >> >>> > and
>> >> >> >> >> >>> > will
>> >> >> >> >> >>> > affect if someone wants to choose mongo as their primary
>> >> >> >> >> >>> > store
>> >> >> >> >> >>> > or
>> >> >> >> >> >>> > not.
>> >> >> >> >> >>> > To
>> >> >> >> >> >>> > not have sharding is one thing, but to promise it and
>> >> >> >> >> >>> > being
>> >> >> >> >> >>> > stuck
>> >> >> >> >> >>> > in
>> >> >> >> >> >>> > production is another.
>> >> >> >> >> >>> >
>> >> >> >> >> >>> > I really thank you guys and appreciate the
>> >> >> >> >> >>> > responsiveness
>> >> >> >> >> >>> > so
>> >> >> >> >> >>> > far.
>> >> >> >> >> >>> > Happy
>> >> >> >> >> >>> > to
>> >> >> >> >> >>> > work together and hope we can resolve this.
>> >> >> >> >> >>> >
>> >> >> >> >> >>> > Thanks so much,
>> >> >> >> >> >>> > -Prasanna
>> >> >> >> >> >>> >
>> >> >> >> >> >>> >
>> >> >> >> >> >>> > On Tue, May 8, 2012 at 7:02 AM, Dan Pasette
>> >> >> >> >> >>> > <dan-***@public.gmane.org>
>> >> >> >> >> >>> > wrote:
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >> Looks like the docs in that collection
>> >> >> >> >> >>> >> (lal_production.facebook_names)
>> >> >> >> >> >>> >> are
>> >> >> >> >> >>> >> small and likely distributed throughout the disk.
>> >> >> >> >> >>> >> Can you check your disk readahead setting?
>> >> >> >> >> >>> >> Please run and post the results of blockev --report.
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >> On Tue, May 8, 2012 at 7:56 AM, Prasanna .S
>> >> >> >> >> >>> >> <myprasanna-***@public.gmane.org>
>> >> >> >> >> >>> >> wrote:
>> >> >> >> >> >>> >>>
>> >> >> >> >> >>> >>> Yes, indeed.
>> >> >> >> >> >>> >>>
>> >> >> >> >> >>> >>> Username: arnold-***@public.gmane.org
>> >> >> >> >> >>> >>> Machine: mongo55
>> >> >> >> >> >>> >>>
>> >> >> >> >> >>> >>>
>> >> >> >> >> >>> >>> On Tue, May 8, 2012 at 4:49 AM, Eliot Horowitz
>> >> >> >> >> >>> >>> <eliot-***@public.gmane.org>
>> >> >> >> >> >>> >>> wrote:
>> >> >> >> >> >>> >>>>
>> >> >> >> >> >>> >>>> Are these machines in MMS?
>> >> >> >> >> >>> >>>> Hard to see much in the pasted stats.
>> >> >> >> >> >>> >>>>
>> >> >> >> >> >>> >>>>
>> >> >> >> >> >>> >>>>
>> >> >> >> >> >>> >>>> On Tue, May 8, 2012 at 5:03 AM, Prasanna .S
>> >> >> >> >> >>> >>>> <myprasanna-***@public.gmane.org>
>> >> >> >> >> >>> >>>> wrote:
>> >> >> >> >> >>> >>>> > Raid 10 on 8 disks.
>> >> >> >> >> >>> >>>> >
>> >> >> >> >> >>> >>>> > MongoDB shell version: 2.0.4
>> >> >> >> >> >>> >>>> > Tue May 8 09:02:29 [initandlisten] db version
>> >> >> >> >> >>> >>>> > v2.0.4,
>> >> >> >> >> >>> >>>> > pdfile
>> >> >> >> >> >>> >>>> > version
>> >> >> >> >> >>> >>>> > 4.5
>> >> >> >> >> >>> >>>> >
>> >> >> >> >> >>> >>>> >
>> >> >> >> >> >>> >>>> > On Tue, May 8, 2012 at 1:49 AM, Eliot Horowitz
>> >> >> >> >> >>> >>>> > <eliot-***@public.gmane.org>
>> >> >> >> >> >>> >>>> > wrote:
>> >> >> >> >> >>> >>>> >>
>> >> >> >> >> >>> >>>> >> What version of mongo and what kind of ebs raid?
>> >> >> >> >> >>> >>>> >>
>> >> >> >> >> >>> >>>> >> On Tue, May 8, 2012 at 4:46 AM, Prasanna
>> >> >> >> >> >>> >>>> >> <myprasanna-***@public.gmane.org>
>> >> >> >> >> >>> >>>> >> wrote:
>> >> >> >> >> >>> >>>> >> > And yes, disabling the balancer, brings the
>> >> >> >> >> >>> >>>> >> > write
>> >> >> >> >> >>> >>>> >> > rate
>> >> >> >> >> >>> >>>> >> > back
>> >> >> >> >> >>> >>>> >> > to
>> >> >> >> >> >>> >>>> >> > zero.
>> >> >> >> >> >>> >>>> >> > Although after 4 minutes!
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> > On Monday, May 7, 2012 11:24:22 PM UTC-7, Nat
>> >> >> >> >> >>> >>>> >> > wrote:
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> I don't think it goes into deadlock. Can you
>> >> >> >> >> >>> >>>> >> >> try
>> >> >> >> >> >>> >>>> >> >> disabling
>> >> >> >> >> >>> >>>> >> >> the
>> >> >> >> >> >>> >>>> >> >> balancer
>> >> >> >> >> >>> >>>> >> >> and see whether the disk write rate going down
>> >> >> >> >> >>> >>>> >> >> or
>> >> >> >> >> >>> >>>> >> >> not.
>> >> >> >> >> >>> >>>> >> >> ________________________________
>> >> >> >> >> >>> >>>> >> >> From: "Prasanna .S" <myprasanna-***@public.gmane.org>
>> >> >> >> >> >>> >>>> >> >> Sender: mongodb-user-/***@public.gmane.org
>> >> >> >> >> >>> >>>> >> >> Date: Mon, 7 May 2012 23:22:11 -0700
>> >> >> >> >> >>> >>>> >> >> To: <mongodb-user-/***@public.gmane.org>
>> >> >> >> >> >>> >>>> >> >> ReplyTo: mongodb-user-/***@public.gmane.org
>> >> >> >> >> >>> >>>> >> >> Cc: Surendran Mahendran<surenspost-***@public.gmane.org>;
>> >> >> >> >> >>> >>>> >> >> Hrushikesh
>> >> >> >> >> >>> >>>> >> >> Tilak<hrushikesh.tilak-***@public.gmane.org>
>> >> >> >> >> >>> >>>> >> >> Subject: [mongodb-user] Mongo sharding deadlock
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> Hey guys,
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> We sharded mongo as per the docs. The issue is
>> >> >> >> >> >>> >>>> >> >> the
>> >> >> >> >> >>> >>>> >> >> balancer
>> >> >> >> >> >>> >>>> >> >> is
>> >> >> >> >> >>> >>>> >> >> not
>> >> >> >> >> >>> >>>> >> >> being
>> >> >> >> >> >>> >>>> >> >> very clever and not throttling at all. Now we
>> >> >> >> >> >>> >>>> >> >> are
>> >> >> >> >> >>> >>>> >> >> stuck
>> >> >> >> >> >>> >>>> >> >> with a
>> >> >> >> >> >>> >>>> >> >> half-sharded
>> >> >> >> >> >>> >>>> >> >> setup and we can't turn on the balancer because
>> >> >> >> >> >>> >>>> >> >> it'd
>> >> >> >> >> >>> >>>> >> >> bring
>> >> >> >> >> >>> >>>> >> >> the
>> >> >> >> >> >>> >>>> >> >> site
>> >> >> >> >> >>> >>>> >> >> down.
>> >> >> >> >> >>> >>>> >> >> All read queries timeout when the disk write
>> >> >> >> >> >>> >>>> >> >> rate
>> >> >> >> >> >>> >>>> >> >> is
>> >> >> >> >> >>> >>>> >> >> 100%.
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> Is there a way out of this mess at all? From a
>> >> >> >> >> >>> >>>> >> >> high-level,
>> >> >> >> >> >>> >>>> >> >> mongo
>> >> >> >> >> >>> >>>> >> >> could
>> >> >> >> >> >>> >>>> >> >> have handled this by any of the following
>> >> >> >> >> >>> >>>> >> >> means:
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> 1. Replicate from the slaves. Mysql does this.
>> >> >> >> >> >>> >>>> >> >> 2. Throttle and replicate only as long as the
>> >> >> >> >> >>> >>>> >> >> disk
>> >> >> >> >> >>> >>>> >> >> util
>> >> >> >> >> >>> >>>> >> >> is
>> >> >> >> >> >>> >>>> >> >> <
>> >> >> >> >> >>> >>>> >> >> 50%
>> >> >> >> >> >>> >>>> >> >> or so.
>> >> >> >> >> >>> >>>> >> >> 3. Have the moveChunk interruptible. If reads
>> >> >> >> >> >>> >>>> >> >> and
>> >> >> >> >> >>> >>>> >> >> writes
>> >> >> >> >> >>> >>>> >> >> come
>> >> >> >> >> >>> >>>> >> >> in-between,
>> >> >> >> >> >>> >>>> >> >> consider them as higher priority, so that a
>> >> >> >> >> >>> >>>> >> >> read/write
>> >> >> >> >> >>> >>>> >> >> is
>> >> >> >> >> >>> >>>> >> >> never
>> >> >> >> >> >>> >>>> >> >> waiting
>> >> >> >> >> >>> >>>> >> >> for
>> >> >> >> >> >>> >>>> >> >> more than a second.
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> Have you guys heard of this problem before?
>> >> >> >> >> >>> >>>> >> >> Because
>> >> >> >> >> >>> >>>> >> >> this
>> >> >> >> >> >>> >>>> >> >> really
>> >> >> >> >> >>> >>>> >> >> make
>> >> >> >> >> >>> >>>> >> >> sharding impossible. I'm not sure what others
>> >> >> >> >> >>> >>>> >> >> are
>> >> >> >> >> >>> >>>> >> >> doing
>> >> >> >> >> >>> >>>> >> >> in
>> >> >> >> >> >>> >>>> >> >> production
>> >> >> >> >> >>> >>>> >> >> to
>> >> >> >> >> >>> >>>> >> >> work around this.
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> Thanks,
>> >> >> >> >> >>> >>>> >> >> -Prasanna
>> >> >> >> >> >>> >>>> >> >> Co-Founder & CTO, lal.com
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> --
>> >> >> >> >> >>> >>>> >> >> You received this message because you are
>> >> >> >> >> >>> >>>> >> >> subscribed
>> >> >> >> >> >>> >>>> >> >> to
>> >> >> >> >> >>> >>>> >> >> the
>> >> >> >> >> >>> >>>> >> >> Google
>> >> >> >> >> >>> >>>> >> >> Groups
>> >> >> >> >> >>> >>>> >> >> "mongodb-user" group.
>> >> >> >> >> >>> >>>> >> >> To post to this group, send email to
>> >> >> >> >> >>> >>>> >> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> >> >> To unsubscribe from this group, send email to
>> >> >> >> >> >>> >>>> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> >> >> For more options, visit this group at
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> > --
>> >> >> >> >> >>> >>>> >> > You received this message because you are
>> >> >> >> >> >>> >>>> >> > subscribed
>> >> >> >> >> >>> >>>> >> > to
>> >> >> >> >> >>> >>>> >> > the
>> >> >> >> >> >>> >>>> >> > Google
>> >> >> >> >> >>> >>>> >> > Groups
>> >> >> >> >> >>> >>>> >> > "mongodb-user" group.
>> >> >> >> >> >>> >>>> >> > To view this discussion on the web visit
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> > https://groups.google.com/d/msg/mongodb-user/-/ePw_QsFo6TYJ.
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> > To post to this group, send email to
>> >> >> >> >> >>> >>>> >> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> >> > To unsubscribe from this group, send email to
>> >> >> >> >> >>> >>>> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> >> > For more options, visit this group at
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >>>> >>
>> >> >> >> >> >>> >>>> >> --
>> >> >> >> >> >>> >>>> >> You received this message because you are
>> >> >> >> >> >>> >>>> >> subscribed
>> >> >> >> >> >>> >>>> >> to
>> >> >> >> >> >>> >>>> >> the
>> >> >> >> >> >>> >>>> >> Google
>> >> >> >> >> >>> >>>> >> Groups
>> >> >> >> >> >>> >>>> >> "mongodb-user" group.
>> >> >> >> >> >>> >>>> >> To post to this group, send email to
>> >> >> >> >> >>> >>>> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> >> To unsubscribe from this group, send email to
>> >> >> >> >> >>> >>>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> >> For more options, visit this group at
>> >> >> >> >> >>> >>>> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >>>> >>
>> >> >> >> >> >>> >>>> >
>> >> >> >> >> >>> >>>> > --
>> >> >> >> >> >>> >>>> > You received this message because you are
>> >> >> >> >> >>> >>>> > subscribed
>> >> >> >> >> >>> >>>> > to
>> >> >> >> >> >>> >>>> > the
>> >> >> >> >> >>> >>>> > Google
>> >> >> >> >> >>> >>>> > Groups
>> >> >> >> >> >>> >>>> > "mongodb-user" group.
>> >> >> >> >> >>> >>>> > To post to this group, send email to
>> >> >> >> >> >>> >>>> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> > To unsubscribe from this group, send email to
>> >> >> >> >> >>> >>>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> > For more options, visit this group at
>> >> >> >> >> >>> >>>> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >>>>
>> >> >> >> >> >>> >>>> --
>> >> >> >> >> >>> >>>> You received this message because you are subscribed
>> >> >> >> >> >>> >>>> to
>> >> >> >> >> >>> >>>> the
>> >> >> >> >> >>> >>>> Google
>> >> >> >> >> >>> >>>> Groups "mongodb-user" group.
>> >> >> >> >> >>> >>>> To post to this group, send email to
>> >> >> >> >> >>> >>>> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> To unsubscribe from this group, send email to
>> >> >> >> >> >>> >>>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> For more options, visit this group at
>> >> >> >> >> >>> >>>> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >>>>
>> >> >> >> >> >>> >>>
>> >> >> >> >> >>> >>> --
>> >> >> >> >> >>> >>> You received this message because you are subscribed
>> >> >> >> >> >>> >>> to
>> >> >> >> >> >>> >>> the
>> >> >> >> >> >>> >>> Google
>> >> >> >> >> >>> >>> Groups
>> >> >> >> >> >>> >>> "mongodb-user" group.
>> >> >> >> >> >>> >>> To post to this group, send email to
>> >> >> >> >> >>> >>> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>> To unsubscribe from this group, send email to
>> >> >> >> >> >>> >>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>> For more options, visit this group at
>> >> >> >> >> >>> >>> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >> --
>> >> >> >> >> >>> >> You received this message because you are subscribed to
>> >> >> >> >> >>> >> the
>> >> >> >> >> >>> >> Google
>> >> >> >> >> >>> >> Groups
>> >> >> >> >> >>> >> "mongodb-user" group.
>> >> >> >> >> >>> >> To post to this group, send email to
>> >> >> >> >> >>> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >> To unsubscribe from this group, send email to
>> >> >> >> >> >>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >> For more options, visit this group at
>> >> >> >> >> >>> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >
>> >> >> >> >> >>> >
>> >> >> >> >> >>> > --
>> >> >> >> >> >>> > You received this message because you are subscribed to
>> >> >> >> >> >>> > the
>> >> >> >> >> >>> > Google
>> >> >> >> >> >>> > Groups
>> >> >> >> >> >>> > "mongodb-user" group.
>> >> >> >> >> >>> > To post to this group, send email to
>> >> >> >> >> >>> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> > To unsubscribe from this group, send email to
>> >> >> >> >> >>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> > For more options, visit this group at
>> >> >> >> >> >>> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>>
>> >> >> >> >> >>> --
>> >> >> >> >> >>> You received this message because you are subscribed to
>> >> >> >> >> >>> the
>> >> >> >> >> >>> Google
>> >> >> >> >> >>> Groups
>> >> >> >> >> >>> "mongodb-user" group.
>> >> >> >> >> >>> To post to this group, send email to
>> >> >> >> >> >>> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> To unsubscribe from this group, send email to
>> >> >> >> >> >>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> For more options, visit this group at
>> >> >> >> >> >>> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>>
>> >> >> >> >> >>
>> >> >> >> >> >
>> >> >> >> >> > --
>> >> >> >> >> > You received this message because you are subscribed to the
>> >> >> >> >> > Google
>> >> >> >> >> > Groups
>> >> >> >> >> > "mongodb-user" group.
>> >> >> >> >> > To post to this group, send email to
>> >> >> >> >> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> > To unsubscribe from this group, send email to
>> >> >> >> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> > For more options, visit this group at
>> >> >> >> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >>
>> >> >> >> >> --
>> >> >> >> >> You received this message because you are subscribed to the
>> >> >> >> >> Google
>> >> >> >> >> Groups
>> >> >> >> >> "mongodb-user" group.
>> >> >> >> >> To post to this group, send email to
>> >> >> >> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> To unsubscribe from this group, send email to
>> >> >> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> For more options, visit this group at
>> >> >> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >>
>> >> >> >> >
>> >> >> >> > --
>> >> >> >> > You received this message because you are subscribed to the
>> >> >> >> > Google
>> >> >> >> > Groups
>> >> >> >> > "mongodb-user" group.
>> >> >> >> > To post to this group, send email to
>> >> >> >> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> > To unsubscribe from this group, send email to
>> >> >> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> > For more options, visit this group at
>> >> >> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >>
>> >> >> >> --
>> >> >> >> You received this message because you are subscribed to the
>> >> >> >> Google
>> >> >> >> Groups
>> >> >> >> "mongodb-user" group.
>> >> >> >> To post to this group, send email to
>> >> >> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> To unsubscribe from this group, send email to
>> >> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> For more options, visit this group at
>> >> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >>
>> >> >> >
>> >> >> > --
>> >> >> > You received this message because you are subscribed to the Google
>> >> >> > Groups
>> >> >> > "mongodb-user" group.
>> >> >> > To post to this group, send email to
>> >> >> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> > To unsubscribe from this group, send email to
>> >> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> > For more options, visit this group at
>> >> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >>
>> >> >> --
>> >> >> You received this message because you are subscribed to the Google
>> >> >> Groups
>> >> >> "mongodb-user" group.
>> >> >> To post to this group, send email to mongodb-user-/***@public.gmane.org.
>> >> >> To unsubscribe from this group, send email to
>> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> For more options, visit this group at
>> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >>
>> >> >
>> >> > --
>> >> > You received this message because you are subscribed to the Google
>> >> > Groups
>> >> > "mongodb-user" group.
>> >> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> > To unsubscribe from this group, send email to
>> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> > For more options, visit this group at
>> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >>
>> >> --
>> >> You received this message because you are subscribed to the Google
>> >> Groups
>> >> "mongodb-user" group.
>> >> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> To unsubscribe from this group, send email to
>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> For more options, visit this group at
>> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >>
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "mongodb-user" group.
>> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> > To unsubscribe from this group, send email to
>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> > For more options, visit this group at
>> > http://groups.google.com/group/mongodb-user?hl=en.
>
>
> On Thursday, May 10, 2012 5:04:33 PM UTC+2, Scott Hernandez wrote:
>>
>> You seem to looking at this as a binary result. Slow reads do not mean
>> downtime. There are many use-cases where many times performance
>> degradation during maintenance windows is acceptable but downtime
>> (stopping reads) is not.
>>
>> We are working with everyone to make sure they can get the best
>> performance and uptime for their systems. If you have more specific
>> areas where you are seeing issue, please create a new thread/post or
>> jira issue with the problem (or vote/watch for existing ones).
>>
>> Nobody thinks the system is perfect as is (how could it be when there
>> are so many different use cases), and there are lots of improvements,
>> fixes and changes going in to address these areas for improvement.
>>
>>
>> On Thu, May 10, 2012 at 7:44 AM, Prasanna .S <myprasanna-***@public.gmane.org> wrote:
>> > That use case can easily be supported by just shutting off the reads
>> > from
>> > the application. Basically they are anyways taking downtime -- that
>> > doesn't
>> > take special support, does it?
>> >
>> > In the normal case, don't you think the reads are the highest priority?
>> > Come
>> > on, this is production.. And I'm telling you, your top clients have this
>> > problem. Whats worse for a DB than downtime?
>> >
>> >
>> > On Thu, May 10, 2012 at 7:37 AM, Scott Hernandez
>> > <scotthernandez-***@public.gmane.org>
>> > wrote:
>> >>
>> >> A simple example is a system where they want the balancer to run as
>> >> quickly as possible at low traffic times to balancer data and don't
>> >> care about the impact of (user) reads. This can be a big deal when you
>> >> are already under-provisioned and need to add a new shard to handle
>> >> users more effectively -- it has come up a few times.
>> >>
>> >> On Thu, May 10, 2012 at 7:33 AM, Prasanna .S <myprasanna-***@public.gmane.org>
>> >> wrote:
>> >> > Scott,
>> >> >
>> >> >> While it might be paramount for some situations/use-cases to
>> >> >> complete all
>> >> >> reads before a move starts, it might not be for others (which I
>> >> > have seen).
>> >> >
>> >> > Can you please explain the "might not be for others" part? Can you
>> >> > give
>> >> > me
>> >> > an example?
>> >> >
>> >> >
>> >> > On Thu, May 10, 2012 at 7:19 AM, Scott Hernandez
>> >> > <scotthernandez-***@public.gmane.org>
>> >> > wrote:
>> >> >>
>> >> >> There are lots of ways to look at this with some competing goals.
>> >> >> While it might be paramount for some situations/use-cases to
>> >> >> complete
>> >> >> all reads before a move starts, it might not be for others (which I
>> >> >> have seen). This fix will allow reads (and other writes) to get
>> >> >> processes while chunks moves happen, period. The issue of locking,
>> >> >> and
>> >> >> contentions on those locks, is very important and there are lots of
>> >> >> improvements being made which help across the boards.
>> >> >>
>> >> >> Feel free to add comments or create new issues with your ideas of
>> >> >> how
>> >> >> your system would like things to best behave:
>> >> >> http://jira.mongodb.org/browse/server
>> >> >>
>> >> >> On Wed, May 9, 2012 at 7:46 PM, Prasanna .S <myprasanna-***@public.gmane.org>
>> >> >> wrote:
>> >> >> > Scott,
>> >> >> >
>> >> >> > The below are heuristics, and not guarantees. What you want from a
>> >> >> > DB
>> >> >> > stand-point is the latter.
>> >> >> >
>> >> >> > The delay between moves, won't provide a guarantee, unless you
>> >> >> > make
>> >> >> > sure
>> >> >> > that
>> >> >> > (1) the next iteration of moveChunk, does not start unless all
>> >> >> > pending
>> >> >> > reads
>> >> >> > complete.
>> >> >> > (2) Each iteration takes less than 1second.
>> >> >> >
>> >> >> > Thoughts?
>> >> >> >
>> >> >> >
>> >> >> > On Wed, May 9, 2012 at 6:18 PM, Scott Hernandez
>> >> >> > <scotthernandez-***@public.gmane.org>
>> >> >> > wrote:
>> >> >> >>
>> >> >> >> The logic there is to do each document write, during the migrate,
>> >> >> >> in
>> >> >> >> a
>> >> >> >> separate lock so other operations can be interleaved. We have
>> >> >> >> seen
>> >> >> >> this change dramatically reduce the constant lock percentage and
>> >> >> >> allow
>> >> >> >> reads, and other writes through during a initial move.
>> >> >> >>
>> >> >> >> There is already a delay between moves which doesn't seem to be
>> >> >> >> your
>> >> >> >> problem.
>> >> >> >>
>> >> >> >> On Wed, May 9, 2012 at 5:37 PM, Prasanna .S
>> >> >> >> <myprasanna-***@public.gmane.org>
>> >> >> >> wrote:
>> >> >> >> > Scott,
>> >> >> >> >
>> >> >> >> > I tried to understand what's going on there. But with a quick
>> >> >> >> > glance,
>> >> >> >> > I
>> >> >> >> > assume, a lock was being held longer than necessary and was
>> >> >> >> > fixed.
>> >> >> >> > It's
>> >> >> >> > unclear how this can help me. The only way to reliably make
>> >> >> >> > sure
>> >> >> >> > the
>> >> >> >> > site
>> >> >> >> > keeps running while a moveChunk is executing, is if, you yield
>> >> >> >> > every
>> >> >> >> > second
>> >> >> >> > within moveChunk and make sure all pending reads complete,
>> >> >> >> > before
>> >> >> >> > the
>> >> >> >> > next
>> >> >> >> > moveChunk period starts.
>> >> >> >> >
>> >> >> >> > Are you guys doing it, or planning to do it? Is there any other
>> >> >> >> > high-level
>> >> >> >> > way to guarantee the reads can be answered within a couple of
>> >> >> >> > seconds?
>> >> >> >> >
>> >> >> >> > Thanks
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > On Wed, May 9, 2012 at 5:17 PM, Scott Hernandez
>> >> >> >> > <scotthernandez-***@public.gmane.org>
>> >> >> >> > wrote:
>> >> >> >> >>
>> >> >> >> >> Okay, looking at the receiving shard mongo55, and donor shard
>> >> >> >> >> mongo43,
>> >> >> >> >> it look like the fix in 2.0.6 (due out in the next two weeks
>> >> >> >> >> of
>> >> >> >> >> so)
>> >> >> >> >> will help you out a lot.
>> >> >> >> >>
>> >> >> >> >> On Wed, May 9, 2012 at 4:37 PM, Prasanna .S
>> >> >> >> >> <myprasanna-***@public.gmane.org>
>> >> >> >> >> wrote:
>> >> >> >> >> > 4:37 the balancer has started and disk shot up.
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > On Wed, May 9, 2012 at 4:30 PM, Prasanna .S
>> >> >> >> >> > <myprasanna-***@public.gmane.org>
>> >> >> >> >> > wrote:
>> >> >> >> >> >>
>> >> >> >> >> >> Hey Scott,
>> >> >> >> >> >>
>> >> >> >> >> >> I'm going to do this now. Will start it and stop it quickly
>> >> >> >> >> >> and
>> >> >> >> >> >> give
>> >> >> >> >> >> you
>> >> >> >> >> >> the exact times.
>> >> >> >> >> >>
>> >> >> >> >> >> Thanks,
>> >> >> >> >> >> -Prasanna
>> >> >> >> >> >>
>> >> >> >> >> >>
>> >> >> >> >> >> On Wed, May 9, 2012 at 3:41 PM, Scott Hernandez
>> >> >> >> >> >> <scotthernandez-***@public.gmane.org>
>> >> >> >> >> >> wrote:
>> >> >> >> >> >>>
>> >> >> >> >> >>> Those values are in blocks (512 bytes), not MB.
>> >> >> >> >> >>>
>> >> >> >> >> >>> We have an update in the upcoming 2.0.6 stable release
>> >> >> >> >> >>> which
>> >> >> >> >> >>> will
>> >> >> >> >> >>> help
>> >> >> >> >> >>> with migrations a bit.
>> >> >> >> >> >>> https://jira.mongodb.org/browse/SERVER-5303
>> >> >> >> >> >>> You
>> >> >> >> >> >>> can click on the version like to see other important
>> >> >> >> >> >>> fixes.
>> >> >> >> >> >>>
>> >> >> >> >> >>> Can you do a few migrates tonight so we can see the
>> >> >> >> >> >>> stats/behavior
>> >> >> >> >> >>> in
>> >> >> >> >> >>> MMS? Please let us know when you do this so we can take a
>> >> >> >> >> >>> look
>> >> >> >> >> >>> at
>> >> >> >> >> >>> the
>> >> >> >> >> >>> data to see if this fix will help.
>> >> >> >> >> >>>
>> >> >> >> >> >>>
>> >> >> >> >> >>> On Tue, May 8, 2012 at 11:49 AM, Prasanna .S
>> >> >> >> >> >>> <myprasanna-***@public.gmane.org>
>> >> >> >> >> >>> wrote:
>> >> >> >> >> >>> > Hey Dan,
>> >> >> >> >> >>> >
>> >> >> >> >> >>> > Read ahead settings are the defaults (256
>> >> >> >> >> >>> > MB): http://pastebin.com/MHc8s2fq
>> >> >> >> >> >>> > Let me know if you need something else.
>> >> >> >> >> >>> >
>> >> >> >> >> >>> > @Elliot,
>> >> >> >> >> >>> >
>> >> >> >> >> >>> > I spoke with some of my friends at foursquare and it
>> >> >> >> >> >>> > turns
>> >> >> >> >> >>> > out
>> >> >> >> >> >>> > they
>> >> >> >> >> >>> > face the
>> >> >> >> >> >>> > same issue as well. I've had my other friends who run
>> >> >> >> >> >>> > companies >
>> >> >> >> >> >>> > $100M
>> >> >> >> >> >>> > market cap (whom I'd rather not name in this list), tell
>> >> >> >> >> >>> > me
>> >> >> >> >> >>> > about
>> >> >> >> >> >>> > their
>> >> >> >> >> >>> > mongo problems in the past few days. This is pretty
>> >> >> >> >> >>> > serious
>> >> >> >> >> >>> > stuff
>> >> >> >> >> >>> > and
>> >> >> >> >> >>> > will
>> >> >> >> >> >>> > affect if someone wants to choose mongo as their primary
>> >> >> >> >> >>> > store
>> >> >> >> >> >>> > or
>> >> >> >> >> >>> > not.
>> >> >> >> >> >>> > To
>> >> >> >> >> >>> > not have sharding is one thing, but to promise it and
>> >> >> >> >> >>> > being
>> >> >> >> >> >>> > stuck
>> >> >> >> >> >>> > in
>> >> >> >> >> >>> > production is another.
>> >> >> >> >> >>> >
>> >> >> >> >> >>> > I really thank you guys and appreciate the
>> >> >> >> >> >>> > responsiveness
>> >> >> >> >> >>> > so
>> >> >> >> >> >>> > far.
>> >> >> >> >> >>> > Happy
>> >> >> >> >> >>> > to
>> >> >> >> >> >>> > work together and hope we can resolve this.
>> >> >> >> >> >>> >
>> >> >> >> >> >>> > Thanks so much,
>> >> >> >> >> >>> > -Prasanna
>> >> >> >> >> >>> >
>> >> >> >> >> >>> >
>> >> >> >> >> >>> > On Tue, May 8, 2012 at 7:02 AM, Dan Pasette
>> >> >> >> >> >>> > <dan-***@public.gmane.org>
>> >> >> >> >> >>> > wrote:
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >> Looks like the docs in that collection
>> >> >> >> >> >>> >> (lal_production.facebook_names)
>> >> >> >> >> >>> >> are
>> >> >> >> >> >>> >> small and likely distributed throughout the disk.
>> >> >> >> >> >>> >> Can you check your disk readahead setting?
>> >> >> >> >> >>> >> Please run and post the results of blockev --report.
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >> On Tue, May 8, 2012 at 7:56 AM, Prasanna .S
>> >> >> >> >> >>> >> <myprasanna-***@public.gmane.org>
>> >> >> >> >> >>> >> wrote:
>> >> >> >> >> >>> >>>
>> >> >> >> >> >>> >>> Yes, indeed.
>> >> >> >> >> >>> >>>
>> >> >> >> >> >>> >>> Username: arnold-***@public.gmane.org
>> >> >> >> >> >>> >>> Machine: mongo55
>> >> >> >> >> >>> >>>
>> >> >> >> >> >>> >>>
>> >> >> >> >> >>> >>> On Tue, May 8, 2012 at 4:49 AM, Eliot Horowitz
>> >> >> >> >> >>> >>> <eliot-***@public.gmane.org>
>> >> >> >> >> >>> >>> wrote:
>> >> >> >> >> >>> >>>>
>> >> >> >> >> >>> >>>> Are these machines in MMS?
>> >> >> >> >> >>> >>>> Hard to see much in the pasted stats.
>> >> >> >> >> >>> >>>>
>> >> >> >> >> >>> >>>>
>> >> >> >> >> >>> >>>>
>> >> >> >> >> >>> >>>> On Tue, May 8, 2012 at 5:03 AM, Prasanna .S
>> >> >> >> >> >>> >>>> <myprasanna-***@public.gmane.org>
>> >> >> >> >> >>> >>>> wrote:
>> >> >> >> >> >>> >>>> > Raid 10 on 8 disks.
>> >> >> >> >> >>> >>>> >
>> >> >> >> >> >>> >>>> > MongoDB shell version: 2.0.4
>> >> >> >> >> >>> >>>> > Tue May 8 09:02:29 [initandlisten] db version
>> >> >> >> >> >>> >>>> > v2.0.4,
>> >> >> >> >> >>> >>>> > pdfile
>> >> >> >> >> >>> >>>> > version
>> >> >> >> >> >>> >>>> > 4.5
>> >> >> >> >> >>> >>>> >
>> >> >> >> >> >>> >>>> >
>> >> >> >> >> >>> >>>> > On Tue, May 8, 2012 at 1:49 AM, Eliot Horowitz
>> >> >> >> >> >>> >>>> > <eliot-***@public.gmane.org>
>> >> >> >> >> >>> >>>> > wrote:
>> >> >> >> >> >>> >>>> >>
>> >> >> >> >> >>> >>>> >> What version of mongo and what kind of ebs raid?
>> >> >> >> >> >>> >>>> >>
>> >> >> >> >> >>> >>>> >> On Tue, May 8, 2012 at 4:46 AM, Prasanna
>> >> >> >> >> >>> >>>> >> <myprasanna-***@public.gmane.org>
>> >> >> >> >> >>> >>>> >> wrote:
>> >> >> >> >> >>> >>>> >> > And yes, disabling the balancer, brings the
>> >> >> >> >> >>> >>>> >> > write
>> >> >> >> >> >>> >>>> >> > rate
>> >> >> >> >> >>> >>>> >> > back
>> >> >> >> >> >>> >>>> >> > to
>> >> >> >> >> >>> >>>> >> > zero.
>> >> >> >> >> >>> >>>> >> > Although after 4 minutes!
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> > On Monday, May 7, 2012 11:24:22 PM UTC-7, Nat
>> >> >> >> >> >>> >>>> >> > wrote:
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> I don't think it goes into deadlock. Can you
>> >> >> >> >> >>> >>>> >> >> try
>> >> >> >> >> >>> >>>> >> >> disabling
>> >> >> >> >> >>> >>>> >> >> the
>> >> >> >> >> >>> >>>> >> >> balancer
>> >> >> >> >> >>> >>>> >> >> and see whether the disk write rate going down
>> >> >> >> >> >>> >>>> >> >> or
>> >> >> >> >> >>> >>>> >> >> not.
>> >> >> >> >> >>> >>>> >> >> ________________________________
>> >> >> >> >> >>> >>>> >> >> From: "Prasanna .S" <myprasanna-***@public.gmane.org>
>> >> >> >> >> >>> >>>> >> >> Sender: mongodb-user-/***@public.gmane.org
>> >> >> >> >> >>> >>>> >> >> Date: Mon, 7 May 2012 23:22:11 -0700
>> >> >> >> >> >>> >>>> >> >> To: <mongodb-user-/***@public.gmane.org>
>> >> >> >> >> >>> >>>> >> >> ReplyTo: mongodb-user-/***@public.gmane.org
>> >> >> >> >> >>> >>>> >> >> Cc: Surendran Mahendran<surenspost-***@public.gmane.org>;
>> >> >> >> >> >>> >>>> >> >> Hrushikesh
>> >> >> >> >> >>> >>>> >> >> Tilak<hrushikesh.tilak-***@public.gmane.org>
>> >> >> >> >> >>> >>>> >> >> Subject: [mongodb-user] Mongo sharding deadlock
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> Hey guys,
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> We sharded mongo as per the docs. The issue is
>> >> >> >> >> >>> >>>> >> >> the
>> >> >> >> >> >>> >>>> >> >> balancer
>> >> >> >> >> >>> >>>> >> >> is
>> >> >> >> >> >>> >>>> >> >> not
>> >> >> >> >> >>> >>>> >> >> being
>> >> >> >> >> >>> >>>> >> >> very clever and not throttling at all. Now we
>> >> >> >> >> >>> >>>> >> >> are
>> >> >> >> >> >>> >>>> >> >> stuck
>> >> >> >> >> >>> >>>> >> >> with a
>> >> >> >> >> >>> >>>> >> >> half-sharded
>> >> >> >> >> >>> >>>> >> >> setup and we can't turn on the balancer because
>> >> >> >> >> >>> >>>> >> >> it'd
>> >> >> >> >> >>> >>>> >> >> bring
>> >> >> >> >> >>> >>>> >> >> the
>> >> >> >> >> >>> >>>> >> >> site
>> >> >> >> >> >>> >>>> >> >> down.
>> >> >> >> >> >>> >>>> >> >> All read queries timeout when the disk write
>> >> >> >> >> >>> >>>> >> >> rate
>> >> >> >> >> >>> >>>> >> >> is
>> >> >> >> >> >>> >>>> >> >> 100%.
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> Is there a way out of this mess at all? From a
>> >> >> >> >> >>> >>>> >> >> high-level,
>> >> >> >> >> >>> >>>> >> >> mongo
>> >> >> >> >> >>> >>>> >> >> could
>> >> >> >> >> >>> >>>> >> >> have handled this by any of the following
>> >> >> >> >> >>> >>>> >> >> means:
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> 1. Replicate from the slaves. Mysql does this.
>> >> >> >> >> >>> >>>> >> >> 2. Throttle and replicate only as long as the
>> >> >> >> >> >>> >>>> >> >> disk
>> >> >> >> >> >>> >>>> >> >> util
>> >> >> >> >> >>> >>>> >> >> is
>> >> >> >> >> >>> >>>> >> >> <
>> >> >> >> >> >>> >>>> >> >> 50%
>> >> >> >> >> >>> >>>> >> >> or so.
>> >> >> >> >> >>> >>>> >> >> 3. Have the moveChunk interruptible. If reads
>> >> >> >> >> >>> >>>> >> >> and
>> >> >> >> >> >>> >>>> >> >> writes
>> >> >> >> >> >>> >>>> >> >> come
>> >> >> >> >> >>> >>>> >> >> in-between,
>> >> >> >> >> >>> >>>> >> >> consider them as higher priority, so that a
>> >> >> >> >> >>> >>>> >> >> read/write
>> >> >> >> >> >>> >>>> >> >> is
>> >> >> >> >> >>> >>>> >> >> never
>> >> >> >> >> >>> >>>> >> >> waiting
>> >> >> >> >> >>> >>>> >> >> for
>> >> >> >> >> >>> >>>> >> >> more than a second.
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> Have you guys heard of this problem before?
>> >> >> >> >> >>> >>>> >> >> Because
>> >> >> >> >> >>> >>>> >> >> this
>> >> >> >> >> >>> >>>> >> >> really
>> >> >> >> >> >>> >>>> >> >> make
>> >> >> >> >> >>> >>>> >> >> sharding impossible. I'm not sure what others
>> >> >> >> >> >>> >>>> >> >> are
>> >> >> >> >> >>> >>>> >> >> doing
>> >> >> >> >> >>> >>>> >> >> in
>> >> >> >> >> >>> >>>> >> >> production
>> >> >> >> >> >>> >>>> >> >> to
>> >> >> >> >> >>> >>>> >> >> work around this.
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> Thanks,
>> >> >> >> >> >>> >>>> >> >> -Prasanna
>> >> >> >> >> >>> >>>> >> >> Co-Founder & CTO, lal.com
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> --
>> >> >> >> >> >>> >>>> >> >> You received this message because you are
>> >> >> >> >> >>> >>>> >> >> subscribed
>> >> >> >> >> >>> >>>> >> >> to
>> >> >> >> >> >>> >>>> >> >> the
>> >> >> >> >> >>> >>>> >> >> Google
>> >> >> >> >> >>> >>>> >> >> Groups
>> >> >> >> >> >>> >>>> >> >> "mongodb-user" group.
>> >> >> >> >> >>> >>>> >> >> To post to this group, send email to
>> >> >> >> >> >>> >>>> >> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> >> >> To unsubscribe from this group, send email to
>> >> >> >> >> >>> >>>> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> >> >> For more options, visit this group at
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> > --
>> >> >> >> >> >>> >>>> >> > You received this message because you are
>> >> >> >> >> >>> >>>> >> > subscribed
>> >> >> >> >> >>> >>>> >> > to
>> >> >> >> >> >>> >>>> >> > the
>> >> >> >> >> >>> >>>> >> > Google
>> >> >> >> >> >>> >>>> >> > Groups
>> >> >> >> >> >>> >>>> >> > "mongodb-user" group.
>> >> >> >> >> >>> >>>> >> > To view this discussion on the web visit
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> > https://groups.google.com/d/msg/mongodb-user/-/ePw_QsFo6TYJ.
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> > To post to this group, send email to
>> >> >> >> >> >>> >>>> >> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> >> > To unsubscribe from this group, send email to
>> >> >> >> >> >>> >>>> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> >> > For more options, visit this group at
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >>>> >>
>> >> >> >> >> >>> >>>> >> --
>> >> >> >> >> >>> >>>> >> You received this message because you are
>> >> >> >> >> >>> >>>> >> subscribed
>> >> >> >> >> >>> >>>> >> to
>> >> >> >> >> >>> >>>> >> the
>> >> >> >> >> >>> >>>> >> Google
>> >> >> >> >> >>> >>>> >> Groups
>> >> >> >> >> >>> >>>> >> "mongodb-user" group.
>> >> >> >> >> >>> >>>> >> To post to this group, send email to
>> >> >> >> >> >>> >>>> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> >> To unsubscribe from this group, send email to
>> >> >> >> >> >>> >>>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> >> For more options, visit this group at
>> >> >> >> >> >>> >>>> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >>>> >>
>> >> >> >> >> >>> >>>> >
>> >> >> >> >> >>> >>>> > --
>> >> >> >> >> >>> >>>> > You received this message because you are
>> >> >> >> >> >>> >>>> > subscribed
>> >> >> >> >> >>> >>>> > to
>> >> >> >> >> >>> >>>> > the
>> >> >> >> >> >>> >>>> > Google
>> >> >> >> >> >>> >>>> > Groups
>> >> >> >> >> >>> >>>> > "mongodb-user" group.
>> >> >> >> >> >>> >>>> > To post to this group, send email to
>> >> >> >> >> >>> >>>> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> > To unsubscribe from this group, send email to
>> >> >> >> >> >>> >>>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> > For more options, visit this group at
>> >> >> >> >> >>> >>>> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >>>>
>> >> >> >> >> >>> >>>> --
>> >> >> >> >> >>> >>>> You received this message because you are subscribed
>> >> >> >> >> >>> >>>> to
>> >> >> >> >> >>> >>>> the
>> >> >> >> >> >>> >>>> Google
>> >> >> >> >> >>> >>>> Groups "mongodb-user" group.
>> >> >> >> >> >>> >>>> To post to this group, send email to
>> >> >> >> >> >>> >>>> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> To unsubscribe from this group, send email to
>> >> >> >> >> >>> >>>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> For more options, visit this group at
>> >> >> >> >> >>> >>>> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >>>>
>> >> >> >> >> >>> >>>
>> >> >> >> >> >>> >>> --
>> >> >> >> >> >>> >>> You received this message because you are subscribed
>> >> >> >> >> >>> >>> to
>> >> >> >> >> >>> >>> the
>> >> >> >> >> >>> >>> Google
>> >> >> >> >> >>> >>> Groups
>> >> >> >> >> >>> >>> "mongodb-user" group.
>> >> >> >> >> >>> >>> To post to this group, send email to
>> >> >> >> >> >>> >>> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>> To unsubscribe from this group, send email to
>> >> >> >> >> >>> >>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>> For more options, visit this group at
>> >> >> >> >> >>> >>> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >> --
>> >> >> >> >> >>> >> You received this message because you are subscribed to
>> >> >> >> >> >>> >> the
>> >> >> >> >> >>> >> Google
>> >> >> >> >> >>> >> Groups
>> >> >> >> >> >>> >> "mongodb-user" group.
>> >> >> >> >> >>> >> To post to this group, send email to
>> >> >> >> >> >>> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >> To unsubscribe from this group, send email to
>> >> >> >> >> >>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >> For more options, visit this group at
>> >> >> >> >> >>> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >
>> >> >> >> >> >>> >
>> >> >> >> >> >>> > --
>> >> >> >> >> >>> > You received this message because you are subscribed to
>> >> >> >> >> >>> > the
>> >> >> >> >> >>> > Google
>> >> >> >> >> >>> > Groups
>> >> >> >> >> >>> > "mongodb-user" group.
>> >> >> >> >> >>> > To post to this group, send email to
>> >> >> >> >> >>> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> > To unsubscribe from this group, send email to
>> >> >> >> >> >>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> > For more options, visit this group at
>> >> >> >> >> >>> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>>
>> >> >> >> >> >>> --
>> >> >> >> >> >>> You received this message because you are subscribed to
>> >> >> >> >> >>> the
>> >> >> >> >> >>> Google
>> >> >> >> >> >>> Groups
>> >> >> >> >> >>> "mongodb-user" group.
>> >> >> >> >> >>> To post to this group, send email to
>> >> >> >> >> >>> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> To unsubscribe from this group, send email to
>> >> >> >> >> >>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> For more options, visit this group at
>> >> >> >> >> >>> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>>
>> >> >> >> >> >>
>> >> >> >> >> >
>> >> >> >> >> > --
>> >> >> >> >> > You received this message because you are subscribed to the
>> >> >> >> >> > Google
>> >> >> >> >> > Groups
>> >> >> >> >> > "mongodb-user" group.
>> >> >> >> >> > To post to this group, send email to
>> >> >> >> >> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> > To unsubscribe from this group, send email to
>> >> >> >> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> > For more options, visit this group at
>> >> >> >> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >>
>> >> >> >> >> --
>> >> >> >> >> You received this message because you are subscribed to the
>> >> >> >> >> Google
>> >> >> >> >> Groups
>> >> >> >> >> "mongodb-user" group.
>> >> >> >> >> To post to this group, send email to
>> >> >> >> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> To unsubscribe from this group, send email to
>> >> >> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> For more options, visit this group at
>> >> >> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >>
>> >> >> >> >
>> >> >> >> > --
>> >> >> >> > You received this message because you are subscribed to the
>> >> >> >> > Google
>> >> >> >> > Groups
>> >> >> >> > "mongodb-user" group.
>> >> >> >> > To post to this group, send email to
>> >> >> >> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> > To unsubscribe from this group, send email to
>> >> >> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> > For more options, visit this group at
>> >> >> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >>
>> >> >> >> --
>> >> >> >> You received this message because you are subscribed to the
>> >> >> >> Google
>> >> >> >> Groups
>> >> >> >> "mongodb-user" group.
>> >> >> >> To post to this group, send email to
>> >> >> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> To unsubscribe from this group, send email to
>> >> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> For more options, visit this group at
>> >> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >>
>> >> >> >
>> >> >> > --
>> >> >> > You received this message because you are subscribed to the Google
>> >> >> > Groups
>> >> >> > "mongodb-user" group.
>> >> >> > To post to this group, send email to
>> >> >> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> > To unsubscribe from this group, send email to
>> >> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> > For more options, visit this group at
>> >> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >>
>> >> >> --
>> >> >> You received this message because you are subscribed to the Google
>> >> >> Groups
>> >> >> "mongodb-user" group.
>> >> >> To post to this group, send email to mongodb-user-/***@public.gmane.org.
>> >> >> To unsubscribe from this group, send email to
>> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> For more options, visit this group at
>> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >>
>> >> >
>> >> > --
>> >> > You received this message because you are subscribed to the Google
>> >> > Groups
>> >> > "mongodb-user" group.
>> >> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> > To unsubscribe from this group, send email to
>> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> > For more options, visit this group at
>> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >>
>> >> --
>> >> You received this message because you are subscribed to the Google
>> >> Groups
>> >> "mongodb-user" group.
>> >> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> To unsubscribe from this group, send email to
>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> For more options, visit this group at
>> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >>
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "mongodb-user" group.
>> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> > To unsubscribe from this group, send email to
>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> > For more options, visit this group at
>> > http://groups.google.com/group/mongodb-user?hl=en.
>
>
> On Thursday, May 10, 2012 5:04:33 PM UTC+2, Scott Hernandez wrote:
>>
>> You seem to looking at this as a binary result. Slow reads do not mean
>> downtime. There are many use-cases where many times performance
>> degradation during maintenance windows is acceptable but downtime
>> (stopping reads) is not.
>>
>> We are working with everyone to make sure they can get the best
>> performance and uptime for their systems. If you have more specific
>> areas where you are seeing issue, please create a new thread/post or
>> jira issue with the problem (or vote/watch for existing ones).
>>
>> Nobody thinks the system is perfect as is (how could it be when there
>> are so many different use cases), and there are lots of improvements,
>> fixes and changes going in to address these areas for improvement.
>>
>>
>> On Thu, May 10, 2012 at 7:44 AM, Prasanna .S <myprasanna-***@public.gmane.org> wrote:
>> > That use case can easily be supported by just shutting off the reads
>> > from
>> > the application. Basically they are anyways taking downtime -- that
>> > doesn't
>> > take special support, does it?
>> >
>> > In the normal case, don't you think the reads are the highest priority?
>> > Come
>> > on, this is production.. And I'm telling you, your top clients have this
>> > problem. Whats worse for a DB than downtime?
>> >
>> >
>> > On Thu, May 10, 2012 at 7:37 AM, Scott Hernandez
>> > <scotthernandez-***@public.gmane.org>
>> > wrote:
>> >>
>> >> A simple example is a system where they want the balancer to run as
>> >> quickly as possible at low traffic times to balancer data and don't
>> >> care about the impact of (user) reads. This can be a big deal when you
>> >> are already under-provisioned and need to add a new shard to handle
>> >> users more effectively -- it has come up a few times.
>> >>
>> >> On Thu, May 10, 2012 at 7:33 AM, Prasanna .S <myprasanna-***@public.gmane.org>
>> >> wrote:
>> >> > Scott,
>> >> >
>> >> >> While it might be paramount for some situations/use-cases to
>> >> >> complete all
>> >> >> reads before a move starts, it might not be for others (which I
>> >> > have seen).
>> >> >
>> >> > Can you please explain the "might not be for others" part? Can you
>> >> > give
>> >> > me
>> >> > an example?
>> >> >
>> >> >
>> >> > On Thu, May 10, 2012 at 7:19 AM, Scott Hernandez
>> >> > <scotthernandez-***@public.gmane.org>
>> >> > wrote:
>> >> >>
>> >> >> There are lots of ways to look at this with some competing goals.
>> >> >> While it might be paramount for some situations/use-cases to
>> >> >> complete
>> >> >> all reads before a move starts, it might not be for others (which I
>> >> >> have seen). This fix will allow reads (and other writes) to get
>> >> >> processes while chunks moves happen, period. The issue of locking,
>> >> >> and
>> >> >> contentions on those locks, is very important and there are lots of
>> >> >> improvements being made which help across the boards.
>> >> >>
>> >> >> Feel free to add comments or create new issues with your ideas of
>> >> >> how
>> >> >> your system would like things to best behave:
>> >> >> http://jira.mongodb.org/browse/server
>> >> >>
>> >> >> On Wed, May 9, 2012 at 7:46 PM, Prasanna .S <myprasanna-***@public.gmane.org>
>> >> >> wrote:
>> >> >> > Scott,
>> >> >> >
>> >> >> > The below are heuristics, and not guarantees. What you want from a
>> >> >> > DB
>> >> >> > stand-point is the latter.
>> >> >> >
>> >> >> > The delay between moves, won't provide a guarantee, unless you
>> >> >> > make
>> >> >> > sure
>> >> >> > that
>> >> >> > (1) the next iteration of moveChunk, does not start unless all
>> >> >> > pending
>> >> >> > reads
>> >> >> > complete.
>> >> >> > (2) Each iteration takes less than 1second.
>> >> >> >
>> >> >> > Thoughts?
>> >> >> >
>> >> >> >
>> >> >> > On Wed, May 9, 2012 at 6:18 PM, Scott Hernandez
>> >> >> > <scotthernandez-***@public.gmane.org>
>> >> >> > wrote:
>> >> >> >>
>> >> >> >> The logic there is to do each document write, during the migrate,
>> >> >> >> in
>> >> >> >> a
>> >> >> >> separate lock so other operations can be interleaved. We have
>> >> >> >> seen
>> >> >> >> this change dramatically reduce the constant lock percentage and
>> >> >> >> allow
>> >> >> >> reads, and other writes through during a initial move.
>> >> >> >>
>> >> >> >> There is already a delay between moves which doesn't seem to be
>> >> >> >> your
>> >> >> >> problem.
>> >> >> >>
>> >> >> >> On Wed, May 9, 2012 at 5:37 PM, Prasanna .S
>> >> >> >> <myprasanna-***@public.gmane.org>
>> >> >> >> wrote:
>> >> >> >> > Scott,
>> >> >> >> >
>> >> >> >> > I tried to understand what's going on there. But with a quick
>> >> >> >> > glance,
>> >> >> >> > I
>> >> >> >> > assume, a lock was being held longer than necessary and was
>> >> >> >> > fixed.
>> >> >> >> > It's
>> >> >> >> > unclear how this can help me. The only way to reliably make
>> >> >> >> > sure
>> >> >> >> > the
>> >> >> >> > site
>> >> >> >> > keeps running while a moveChunk is executing, is if, you yield
>> >> >> >> > every
>> >> >> >> > second
>> >> >> >> > within moveChunk and make sure all pending reads complete,
>> >> >> >> > before
>> >> >> >> > the
>> >> >> >> > next
>> >> >> >> > moveChunk period starts.
>> >> >> >> >
>> >> >> >> > Are you guys doing it, or planning to do it? Is there any other
>> >> >> >> > high-level
>> >> >> >> > way to guarantee the reads can be answered within a couple of
>> >> >> >> > seconds?
>> >> >> >> >
>> >> >> >> > Thanks
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > On Wed, May 9, 2012 at 5:17 PM, Scott Hernandez
>> >> >> >> > <scotthernandez-***@public.gmane.org>
>> >> >> >> > wrote:
>> >> >> >> >>
>> >> >> >> >> Okay, looking at the receiving shard mongo55, and donor shard
>> >> >> >> >> mongo43,
>> >> >> >> >> it look like the fix in 2.0.6 (due out in the next two weeks
>> >> >> >> >> of
>> >> >> >> >> so)
>> >> >> >> >> will help you out a lot.
>> >> >> >> >>
>> >> >> >> >> On Wed, May 9, 2012 at 4:37 PM, Prasanna .S
>> >> >> >> >> <myprasanna-***@public.gmane.org>
>> >> >> >> >> wrote:
>> >> >> >> >> > 4:37 the balancer has started and disk shot up.
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > On Wed, May 9, 2012 at 4:30 PM, Prasanna .S
>> >> >> >> >> > <myprasanna-***@public.gmane.org>
>> >> >> >> >> > wrote:
>> >> >> >> >> >>
>> >> >> >> >> >> Hey Scott,
>> >> >> >> >> >>
>> >> >> >> >> >> I'm going to do this now. Will start it and stop it quickly
>> >> >> >> >> >> and
>> >> >> >> >> >> give
>> >> >> >> >> >> you
>> >> >> >> >> >> the exact times.
>> >> >> >> >> >>
>> >> >> >> >> >> Thanks,
>> >> >> >> >> >> -Prasanna
>> >> >> >> >> >>
>> >> >> >> >> >>
>> >> >> >> >> >> On Wed, May 9, 2012 at 3:41 PM, Scott Hernandez
>> >> >> >> >> >> <scotthernandez-***@public.gmane.org>
>> >> >> >> >> >> wrote:
>> >> >> >> >> >>>
>> >> >> >> >> >>> Those values are in blocks (512 bytes), not MB.
>> >> >> >> >> >>>
>> >> >> >> >> >>> We have an update in the upcoming 2.0.6 stable release
>> >> >> >> >> >>> which
>> >> >> >> >> >>> will
>> >> >> >> >> >>> help
>> >> >> >> >> >>> with migrations a bit.
>> >> >> >> >> >>> https://jira.mongodb.org/browse/SERVER-5303
>> >> >> >> >> >>> You
>> >> >> >> >> >>> can click on the version like to see other important
>> >> >> >> >> >>> fixes.
>> >> >> >> >> >>>
>> >> >> >> >> >>> Can you do a few migrates tonight so we can see the
>> >> >> >> >> >>> stats/behavior
>> >> >> >> >> >>> in
>> >> >> >> >> >>> MMS? Please let us know when you do this so we can take a
>> >> >> >> >> >>> look
>> >> >> >> >> >>> at
>> >> >> >> >> >>> the
>> >> >> >> >> >>> data to see if this fix will help.
>> >> >> >> >> >>>
>> >> >> >> >> >>>
>> >> >> >> >> >>> On Tue, May 8, 2012 at 11:49 AM, Prasanna .S
>> >> >> >> >> >>> <myprasanna-***@public.gmane.org>
>> >> >> >> >> >>> wrote:
>> >> >> >> >> >>> > Hey Dan,
>> >> >> >> >> >>> >
>> >> >> >> >> >>> > Read ahead settings are the defaults (256
>> >> >> >> >> >>> > MB): http://pastebin.com/MHc8s2fq
>> >> >> >> >> >>> > Let me know if you need something else.
>> >> >> >> >> >>> >
>> >> >> >> >> >>> > @Elliot,
>> >> >> >> >> >>> >
>> >> >> >> >> >>> > I spoke with some of my friends at foursquare and it
>> >> >> >> >> >>> > turns
>> >> >> >> >> >>> > out
>> >> >> >> >> >>> > they
>> >> >> >> >> >>> > face the
>> >> >> >> >> >>> > same issue as well. I've had my other friends who run
>> >> >> >> >> >>> > companies >
>> >> >> >> >> >>> > $100M
>> >> >> >> >> >>> > market cap (whom I'd rather not name in this list), tell
>> >> >> >> >> >>> > me
>> >> >> >> >> >>> > about
>> >> >> >> >> >>> > their
>> >> >> >> >> >>> > mongo problems in the past few days. This is pretty
>> >> >> >> >> >>> > serious
>> >> >> >> >> >>> > stuff
>> >> >> >> >> >>> > and
>> >> >> >> >> >>> > will
>> >> >> >> >> >>> > affect if someone wants to choose mongo as their primary
>> >> >> >> >> >>> > store
>> >> >> >> >> >>> > or
>> >> >> >> >> >>> > not.
>> >> >> >> >> >>> > To
>> >> >> >> >> >>> > not have sharding is one thing, but to promise it and
>> >> >> >> >> >>> > being
>> >> >> >> >> >>> > stuck
>> >> >> >> >> >>> > in
>> >> >> >> >> >>> > production is another.
>> >> >> >> >> >>> >
>> >> >> >> >> >>> > I really thank you guys and appreciate the
>> >> >> >> >> >>> > responsiveness
>> >> >> >> >> >>> > so
>> >> >> >> >> >>> > far.
>> >> >> >> >> >>> > Happy
>> >> >> >> >> >>> > to
>> >> >> >> >> >>> > work together and hope we can resolve this.
>> >> >> >> >> >>> >
>> >> >> >> >> >>> > Thanks so much,
>> >> >> >> >> >>> > -Prasanna
>> >> >> >> >> >>> >
>> >> >> >> >> >>> >
>> >> >> >> >> >>> > On Tue, May 8, 2012 at 7:02 AM, Dan Pasette
>> >> >> >> >> >>> > <dan-***@public.gmane.org>
>> >> >> >> >> >>> > wrote:
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >> Looks like the docs in that collection
>> >> >> >> >> >>> >> (lal_production.facebook_names)
>> >> >> >> >> >>> >> are
>> >> >> >> >> >>> >> small and likely distributed throughout the disk.
>> >> >> >> >> >>> >> Can you check your disk readahead setting?
>> >> >> >> >> >>> >> Please run and post the results of blockev --report.
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >> On Tue, May 8, 2012 at 7:56 AM, Prasanna .S
>> >> >> >> >> >>> >> <myprasanna-***@public.gmane.org>
>> >> >> >> >> >>> >> wrote:
>> >> >> >> >> >>> >>>
>> >> >> >> >> >>> >>> Yes, indeed.
>> >> >> >> >> >>> >>>
>> >> >> >> >> >>> >>> Username: arnold-***@public.gmane.org
>> >> >> >> >> >>> >>> Machine: mongo55
>> >> >> >> >> >>> >>>
>> >> >> >> >> >>> >>>
>> >> >> >> >> >>> >>> On Tue, May 8, 2012 at 4:49 AM, Eliot Horowitz
>> >> >> >> >> >>> >>> <eliot-***@public.gmane.org>
>> >> >> >> >> >>> >>> wrote:
>> >> >> >> >> >>> >>>>
>> >> >> >> >> >>> >>>> Are these machines in MMS?
>> >> >> >> >> >>> >>>> Hard to see much in the pasted stats.
>> >> >> >> >> >>> >>>>
>> >> >> >> >> >>> >>>>
>> >> >> >> >> >>> >>>>
>> >> >> >> >> >>> >>>> On Tue, May 8, 2012 at 5:03 AM, Prasanna .S
>> >> >> >> >> >>> >>>> <myprasanna-***@public.gmane.org>
>> >> >> >> >> >>> >>>> wrote:
>> >> >> >> >> >>> >>>> > Raid 10 on 8 disks.
>> >> >> >> >> >>> >>>> >
>> >> >> >> >> >>> >>>> > MongoDB shell version: 2.0.4
>> >> >> >> >> >>> >>>> > Tue May 8 09:02:29 [initandlisten] db version
>> >> >> >> >> >>> >>>> > v2.0.4,
>> >> >> >> >> >>> >>>> > pdfile
>> >> >> >> >> >>> >>>> > version
>> >> >> >> >> >>> >>>> > 4.5
>> >> >> >> >> >>> >>>> >
>> >> >> >> >> >>> >>>> >
>> >> >> >> >> >>> >>>> > On Tue, May 8, 2012 at 1:49 AM, Eliot Horowitz
>> >> >> >> >> >>> >>>> > <eliot-***@public.gmane.org>
>> >> >> >> >> >>> >>>> > wrote:
>> >> >> >> >> >>> >>>> >>
>> >> >> >> >> >>> >>>> >> What version of mongo and what kind of ebs raid?
>> >> >> >> >> >>> >>>> >>
>> >> >> >> >> >>> >>>> >> On Tue, May 8, 2012 at 4:46 AM, Prasanna
>> >> >> >> >> >>> >>>> >> <myprasanna-***@public.gmane.org>
>> >> >> >> >> >>> >>>> >> wrote:
>> >> >> >> >> >>> >>>> >> > And yes, disabling the balancer, brings the
>> >> >> >> >> >>> >>>> >> > write
>> >> >> >> >> >>> >>>> >> > rate
>> >> >> >> >> >>> >>>> >> > back
>> >> >> >> >> >>> >>>> >> > to
>> >> >> >> >> >>> >>>> >> > zero.
>> >> >> >> >> >>> >>>> >> > Although after 4 minutes!
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> > On Monday, May 7, 2012 11:24:22 PM UTC-7, Nat
>> >> >> >> >> >>> >>>> >> > wrote:
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> I don't think it goes into deadlock. Can you
>> >> >> >> >> >>> >>>> >> >> try
>> >> >> >> >> >>> >>>> >> >> disabling
>> >> >> >> >> >>> >>>> >> >> the
>> >> >> >> >> >>> >>>> >> >> balancer
>> >> >> >> >> >>> >>>> >> >> and see whether the disk write rate going down
>> >> >> >> >> >>> >>>> >> >> or
>> >> >> >> >> >>> >>>> >> >> not.
>> >> >> >> >> >>> >>>> >> >> ________________________________
>> >> >> >> >> >>> >>>> >> >> From: "Prasanna .S" <myprasanna-***@public.gmane.org>
>> >> >> >> >> >>> >>>> >> >> Sender: mongodb-user-/***@public.gmane.org
>> >> >> >> >> >>> >>>> >> >> Date: Mon, 7 May 2012 23:22:11 -0700
>> >> >> >> >> >>> >>>> >> >> To: <mongodb-user-/***@public.gmane.org>
>> >> >> >> >> >>> >>>> >> >> ReplyTo: mongodb-user-/***@public.gmane.org
>> >> >> >> >> >>> >>>> >> >> Cc: Surendran Mahendran<surenspost-***@public.gmane.org>;
>> >> >> >> >> >>> >>>> >> >> Hrushikesh
>> >> >> >> >> >>> >>>> >> >> Tilak<hrushikesh.tilak-***@public.gmane.org>
>> >> >> >> >> >>> >>>> >> >> Subject: [mongodb-user] Mongo sharding deadlock
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> Hey guys,
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> We sharded mongo as per the docs. The issue is
>> >> >> >> >> >>> >>>> >> >> the
>> >> >> >> >> >>> >>>> >> >> balancer
>> >> >> >> >> >>> >>>> >> >> is
>> >> >> >> >> >>> >>>> >> >> not
>> >> >> >> >> >>> >>>> >> >> being
>> >> >> >> >> >>> >>>> >> >> very clever and not throttling at all. Now we
>> >> >> >> >> >>> >>>> >> >> are
>> >> >> >> >> >>> >>>> >> >> stuck
>> >> >> >> >> >>> >>>> >> >> with a
>> >> >> >> >> >>> >>>> >> >> half-sharded
>> >> >> >> >> >>> >>>> >> >> setup and we can't turn on the balancer because
>> >> >> >> >> >>> >>>> >> >> it'd
>> >> >> >> >> >>> >>>> >> >> bring
>> >> >> >> >> >>> >>>> >> >> the
>> >> >> >> >> >>> >>>> >> >> site
>> >> >> >> >> >>> >>>> >> >> down.
>> >> >> >> >> >>> >>>> >> >> All read queries timeout when the disk write
>> >> >> >> >> >>> >>>> >> >> rate
>> >> >> >> >> >>> >>>> >> >> is
>> >> >> >> >> >>> >>>> >> >> 100%.
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> Is there a way out of this mess at all? From a
>> >> >> >> >> >>> >>>> >> >> high-level,
>> >> >> >> >> >>> >>>> >> >> mongo
>> >> >> >> >> >>> >>>> >> >> could
>> >> >> >> >> >>> >>>> >> >> have handled this by any of the following
>> >> >> >> >> >>> >>>> >> >> means:
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> 1. Replicate from the slaves. Mysql does this.
>> >> >> >> >> >>> >>>> >> >> 2. Throttle and replicate only as long as the
>> >> >> >> >> >>> >>>> >> >> disk
>> >> >> >> >> >>> >>>> >> >> util
>> >> >> >> >> >>> >>>> >> >> is
>> >> >> >> >> >>> >>>> >> >> <
>> >> >> >> >> >>> >>>> >> >> 50%
>> >> >> >> >> >>> >>>> >> >> or so.
>> >> >> >> >> >>> >>>> >> >> 3. Have the moveChunk interruptible. If reads
>> >> >> >> >> >>> >>>> >> >> and
>> >> >> >> >> >>> >>>> >> >> writes
>> >> >> >> >> >>> >>>> >> >> come
>> >> >> >> >> >>> >>>> >> >> in-between,
>> >> >> >> >> >>> >>>> >> >> consider them as higher priority, so that a
>> >> >> >> >> >>> >>>> >> >> read/write
>> >> >> >> >> >>> >>>> >> >> is
>> >> >> >> >> >>> >>>> >> >> never
>> >> >> >> >> >>> >>>> >> >> waiting
>> >> >> >> >> >>> >>>> >> >> for
>> >> >> >> >> >>> >>>> >> >> more than a second.
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> Have you guys heard of this problem before?
>> >> >> >> >> >>> >>>> >> >> Because
>> >> >> >> >> >>> >>>> >> >> this
>> >> >> >> >> >>> >>>> >> >> really
>> >> >> >> >> >>> >>>> >> >> make
>> >> >> >> >> >>> >>>> >> >> sharding impossible. I'm not sure what others
>> >> >> >> >> >>> >>>> >> >> are
>> >> >> >> >> >>> >>>> >> >> doing
>> >> >> >> >> >>> >>>> >> >> in
>> >> >> >> >> >>> >>>> >> >> production
>> >> >> >> >> >>> >>>> >> >> to
>> >> >> >> >> >>> >>>> >> >> work around this.
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> Thanks,
>> >> >> >> >> >>> >>>> >> >> -Prasanna
>> >> >> >> >> >>> >>>> >> >> Co-Founder & CTO, lal.com
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> --
>> >> >> >> >> >>> >>>> >> >> You received this message because you are
>> >> >> >> >> >>> >>>> >> >> subscribed
>> >> >> >> >> >>> >>>> >> >> to
>> >> >> >> >> >>> >>>> >> >> the
>> >> >> >> >> >>> >>>> >> >> Google
>> >> >> >> >> >>> >>>> >> >> Groups
>> >> >> >> >> >>> >>>> >> >> "mongodb-user" group.
>> >> >> >> >> >>> >>>> >> >> To post to this group, send email to
>> >> >> >> >> >>> >>>> >> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> >> >> To unsubscribe from this group, send email to
>> >> >> >> >> >>> >>>> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> >> >> For more options, visit this group at
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> > --
>> >> >> >> >> >>> >>>> >> > You received this message because you are
>> >> >> >> >> >>> >>>> >> > subscribed
>> >> >> >> >> >>> >>>> >> > to
>> >> >> >> >> >>> >>>> >> > the
>> >> >> >> >> >>> >>>> >> > Google
>> >> >> >> >> >>> >>>> >> > Groups
>> >> >> >> >> >>> >>>> >> > "mongodb-user" group.
>> >> >> >> >> >>> >>>> >> > To view this discussion on the web visit
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> > https://groups.google.com/d/msg/mongodb-user/-/ePw_QsFo6TYJ.
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> > To post to this group, send email to
>> >> >> >> >> >>> >>>> >> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> >> > To unsubscribe from this group, send email to
>> >> >> >> >> >>> >>>> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> >> > For more options, visit this group at
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >>>> >>
>> >> >> >> >> >>> >>>> >> --
>> >> >> >> >> >>> >>>> >> You received this message because you are
>> >> >> >> >> >>> >>>> >> subscribed
>> >> >> >> >> >>> >>>> >> to
>> >> >> >> >> >>> >>>> >> the
>> >> >> >> >> >>> >>>> >> Google
>> >> >> >> >> >>> >>>> >> Groups
>> >> >> >> >> >>> >>>> >> "mongodb-user" group.
>> >> >> >> >> >>> >>>> >> To post to this group, send email to
>> >> >> >> >> >>> >>>> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> >> To unsubscribe from this group, send email to
>> >> >> >> >> >>> >>>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> >> For more options, visit this group at
>> >> >> >> >> >>> >>>> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >>>> >>
>> >> >> >> >> >>> >>>> >
>> >> >> >> >> >>> >>>> > --
>> >> >> >> >> >>> >>>> > You received this message because you are
>> >> >> >> >> >>> >>>> > subscribed
>> >> >> >> >> >>> >>>> > to
>> >> >> >> >> >>> >>>> > the
>> >> >> >> >> >>> >>>> > Google
>> >> >> >> >> >>> >>>> > Groups
>> >> >> >> >> >>> >>>> > "mongodb-user" group.
>> >> >> >> >> >>> >>>> > To post to this group, send email to
>> >> >> >> >> >>> >>>> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> > To unsubscribe from this group, send email to
>> >> >> >> >> >>> >>>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> > For more options, visit this group at
>> >> >> >> >> >>> >>>> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >>>>
>> >> >> >> >> >>> >>>> --
>> >> >> >> >> >>> >>>> You received this message because you are subscribed
>> >> >> >> >> >>> >>>> to
>> >> >> >> >> >>> >>>> the
>> >> >> >> >> >>> >>>> Google
>> >> >> >> >> >>> >>>> Groups "mongodb-user" group.
>> >> >> >> >> >>> >>>> To post to this group, send email to
>> >> >> >> >> >>> >>>> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> To unsubscribe from this group, send email to
>> >> >> >> >> >>> >>>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> For more options, visit this group at
>> >> >> >> >> >>> >>>> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >>>>
>> >> >> >> >> >>> >>>
>> >> >> >> >> >>> >>> --
>> >> >> >> >> >>> >>> You received this message because you are subscribed
>> >> >> >> >> >>> >>> to
>> >> >> >> >> >>> >>> the
>> >> >> >> >> >>> >>> Google
>> >> >> >> >> >>> >>> Groups
>> >> >> >> >> >>> >>> "mongodb-user" group.
>> >> >> >> >> >>> >>> To post to this group, send email to
>> >> >> >> >> >>> >>> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>> To unsubscribe from this group, send email to
>> >> >> >> >> >>> >>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>> For more options, visit this group at
>> >> >> >> >> >>> >>> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >> --
>> >> >> >> >> >>> >> You received this message because you are subscribed to
>> >> >> >> >> >>> >> the
>> >> >> >> >> >>> >> Google
>> >> >> >> >> >>> >> Groups
>> >> >> >> >> >>> >> "mongodb-user" group.
>> >> >> >> >> >>> >> To post to this group, send email to
>> >> >> >> >> >>> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >> To unsubscribe from this group, send email to
>> >> >> >> >> >>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >> For more options, visit this group at
>> >> >> >> >> >>> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >
>> >> >> >> >> >>> >
>> >> >> >> >> >>> > --
>> >> >> >> >> >>> > You received this message because you are subscribed to
>> >> >> >> >> >>> > the
>> >> >> >> >> >>> > Google
>> >> >> >> >> >>> > Groups
>> >> >> >> >> >>> > "mongodb-user" group.
>> >> >> >> >> >>> > To post to this group, send email to
>> >> >> >> >> >>> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> > To unsubscribe from this group, send email to
>> >> >> >> >> >>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> > For more options, visit this group at
>> >> >> >> >> >>> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>>
>> >> >> >> >> >>> --
>> >> >> >> >> >>> You received this message because you are subscribed to
>> >> >> >> >> >>> the
>> >> >> >> >> >>> Google
>> >> >> >> >> >>> Groups
>> >> >> >> >> >>> "mongodb-user" group.
>> >> >> >> >> >>> To post to this group, send email to
>> >> >> >> >> >>> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> To unsubscribe from this group, send email to
>> >> >> >> >> >>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> For more options, visit this group at
>> >> >> >> >> >>> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>>
>> >> >> >> >> >>
>> >> >> >> >> >
>> >> >> >> >> > --
>> >> >> >> >> > You received this message because you are subscribed to the
>> >> >> >> >> > Google
>> >> >> >> >> > Groups
>> >> >> >> >> > "mongodb-user" group.
>> >> >> >> >> > To post to this group, send email to
>> >> >> >> >> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> > To unsubscribe from this group, send email to
>> >> >> >> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> > For more options, visit this group at
>> >> >> >> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >>
>> >> >> >> >> --
>> >> >> >> >> You received this message because you are subscribed to the
>> >> >> >> >> Google
>> >> >> >> >> Groups
>> >> >> >> >> "mongodb-user" group.
>> >> >> >> >> To post to this group, send email to
>> >> >> >> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> To unsubscribe from this group, send email to
>> >> >> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> For more options, visit this group at
>> >> >> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >>
>> >> >> >> >
>> >> >> >> > --
>> >> >> >> > You received this message because you are subscribed to the
>> >> >> >> > Google
>> >> >> >> > Groups
>> >> >> >> > "mongodb-user" group.
>> >> >> >> > To post to this group, send email to
>> >> >> >> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> > To unsubscribe from this group, send email to
>> >> >> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> > For more options, visit this group at
>> >> >> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >>
>> >> >> >> --
>> >> >> >> You received this message because you are subscribed to the
>> >> >> >> Google
>> >> >> >> Groups
>> >> >> >> "mongodb-user" group.
>> >> >> >> To post to this group, send email to
>> >> >> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> To unsubscribe from this group, send email to
>> >> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> For more options, visit this group at
>> >> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >>
>> >> >> >
>> >> >> > --
>> >> >> > You received this message because you are subscribed to the Google
>> >> >> > Groups
>> >> >> > "mongodb-user" group.
>> >> >> > To post to this group, send email to
>> >> >> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> > To unsubscribe from this group, send email to
>> >> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> > For more options, visit this group at
>> >> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >>
>> >> >> --
>> >> >> You received this message because you are subscribed to the Google
>> >> >> Groups
>> >> >> "mongodb-user" group.
>> >> >> To post to this group, send email to mongodb-user-/***@public.gmane.org.
>> >> >> To unsubscribe from this group, send email to
>> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> For more options, visit this group at
>> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >>
>> >> >
>> >> > --
>> >> > You received this message because you are subscribed to the Google
>> >> > Groups
>> >> > "mongodb-user" group.
>> >> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> > To unsubscribe from this group, send email to
>> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> > For more options, visit this group at
>> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >>
>> >> --
>> >> You received this message because you are subscribed to the Google
>> >> Groups
>> >> "mongodb-user" group.
>> >> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> To unsubscribe from this group, send email to
>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> For more options, visit this group at
>> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >>
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "mongodb-user" group.
>> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> > To unsubscribe from this group, send email to
>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> > For more options, visit this group at
>> > http://groups.google.com/group/mongodb-user?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/mongodb-user/-/YRlv9SXrV6QJ.
> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> To unsubscribe from this group, send email to
> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.
--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to mongodb-user+***@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.