[systemd-devel] [PATCH] usb-storage: Add quirk to defeat Kindle's automatic unload

[Alan Stern]

Matthias reports that the Amazon Kindle automatically removes its
emulated media if it doesn't receive another SCSI command within about
one second after a SYNCHRONIZE CACHE. It does so even when the host
has sent a PREVENT MEDIUM REMOVAL command. The reason for this
behavior isn't clear, although it's not hard to make some guesses.

Could Kindle be fixed not to required such workaround? Is there a way
to open a bug with Amazon?

You really should be asking people who work for Amazon. I suspect that
nobody who regularly reads this mailing list knows the answer.

(If you look through the MAINTAINERS file, you'll find there are a few
kernel developers who do work for Amazon, or at least, have @amazon.com
email addresses. Try asking some of them.)

But even if the Kindle firmware gets changed (which I doubt will
happen), we would still want to support all the old devices that aren't
running the updated firmware.

Alan Stern

[Ulrich Windl]

Matthias reports that the Amazon Kindle automatically removes its
emulated media if it doesn't receive another SCSI command within about
one second after a SYNCHRONIZE CACHE. It does so even when the host
has sent a PREVENT MEDIUM REMOVAL command. The reason for this
behavior isn't clear, although it's not hard to make some guesses.

Actually I think Amazon should fix the firmware.

You are free to suggest to them that they change it. Even if you do
find the right people to ask about this, I'd be very surprised if they
agreed to make the change.

It seems the main goal was to prevent the use of open software to manage the
content.

This is guesswork on your part, and I disagree. I think the main goal

Yes, it's guessing, but the real story is: Recently I tried different bluetooth "ear plugs with noise cancelling". Those from Samsung just worked out of the box (regarding audio output at least), while those from Sony did not.
Now you can decide to patch the kernel, or simply buy the other product...

was to improve the user experience by making the Kindle return to normal
operating mode automatically when file transfers are finished, rather
than requiring the user to do something extra. But that's also just a
guess.
Alan Stern

[Lennart Poettering]

Hi,

I think it would be mildly better, but not a whole lot. Since the
Kindle describes itself as having removable media, the kernel normally
probes it periodically to make sure the media remains present. The
default probing interval is 2 seconds. Reducing it to 0.9 seconds
doesn't represent an exorbitant additional load IMO -- especially since
Kindles don't tend to spend huge amounts of time connected to computers.

Ah, I did not know that the default polling interval was that low(ish),
given that the default indeed is already that low, then changing it to
0.8 seconds would not be a big change. And we probably have a lot of
lower hanging fruit for unnecessary wakeups then that.

So we need to make a decision: Should the patch be merged, or should we
punt the issue to userspace tools?
On the plus side, the patch is rather small and non-invasive (although
it does allocate the last remaining bit in the 32-bit fflags field).
There's also the advantage of sending the extra command only when it is
needed, as opposed to increasing the overall frequency of TUR polling.
Any opinions?

I'd argue that this should be done in the kernel.

IIUC the issue can actually lead to data corruption, no? i.e. some
program writes 25 different files to different places on such an fs,
then calls fsync() on one of them but not the others. Then quite
possibly the fsync() will trigger a device disconnect sooner or later
at which point the one file surely hit the disk, but there's no
guarantee for the other 24, they might remain cached in memory and are
never written out.

I'd say quirks that are necessary to avoid data corruption should
better be done in the kernel and udev's hwdb stuff is only for stuff
that "fills in gaps", i.e. adds additional tweaks that make things
prettier, cleaner, nicer, more efficient but not things that make the
basic things work, and data integrity sounds pretty basic to me.

Or to give a counter example: the device advertises it can do media
change, but actually cannot, right, it's not a floppy drive or cdrom
driver after all? maybe hwdb would thus actually be the place for the
opposite of the suggested fix: turn off the media change polling to
reduce needless wakeups.

I mean, I'd be more open to this if this was a frequent thing and the
database for devices like this was just tooo large for the kernel to
carry, but that's not the case here either: it's two devices afaik,
and such an issue wasn't seen elswhere.

Lennart

--
Lennart Poettering, Berlin

[Alan Stern]

in

Post by Lennart Poettering
I'd say quirks that are necessary to avoid data corruption should
better be done in the kernel and udev's hwdb stuff is only for stuff
that "fills in gaps", i.e. adds additional tweaks that make things
prettier, cleaner, nicer, more efficient but not things that make the
basic things work, and data integrity sounds pretty basic to me.

But seeing the list of bad, broken or ill-designed hardware grow year by year,
I wonder whether we really want all that bloat in the kernel.

Post by Lennart Poettering
Or to give a counter example: the device advertises it can do media
change, but actually cannot, right, it's not a floppy drive or cdrom
driver after all? maybe hwdb would thus actually be the place for the
opposite of the suggested fix: turn off the media change polling to
reduce needless wakeups.

I actually think it would be best if those work-arounds could be loadable as
module, and the vendors of broken hardware can provide the modules that
document their broken design as well.

If you can come up with a way to do this (preferably in the form of a
patch), that would be great. I can't think of any way to remove this
information from the kernel.

Alan Stern

[Matthias Schwarzott]

I implemented solution 3b. This is the pullrequest for udev (systemd
https://github.com/systemd/systemd/pull/19002
Now Lennart asks if udev is the best place for such hacks/work-arounds?
Well, I implemented it as suggested by Alan (see above). This was the
simplest of the considered alternatives. Different quirks in kernel has been
considered, but are more effort to be implemented.

Lennart probably isn't aware how the usb-storage driver works. It does
not create commands on its own; it merely sends the commands that it
gets from higher SCSI layers.
It may be possible to modify the SCSI core, to make it send a TEST UNIT
READY command immediately following any SYNCHRONIZE CACHE to a Kindle.
However, there may be an easier solution. usb-storage does indeed send
a command of its own, REQUEST SENSE, to get error data when a command
fails. The patch below will make it do the same thing whenever it sends
a SYNCHRONIZE CACHE to a Kindle, failure or not.
The only question is whether the Kindle will regard REQUEST SENSE as a
sufficient indication that it shouldn't do an eject. The only way to
find out is by testing the patch.

The patch is a lot shorter than I expected it to be.

I tried it. The new udev-rule is commented, so polling interval is back
at 2000 ms.

Testing it:
# cat /sys/block/sde/events_poll_msecs
-1
# cat /sys/module/block/parameters/events_dfl_poll_msecs
2000
# python -c "import time; import os; f = open('/dev/sde1', 'r+b');
os.fsync(f); time.sleep(4)"

This is wireshark output of the test:

85 4.701949 host 3.8.1 USBMS 95 SCSI: Test Unit Ready LUN: 0x00
86 4.701972 3.8.1 host USB 64 URB_BULK out
87 4.701975 host 3.8.1 USB 64 URB_BULK in
88 4.702030 3.8.1 host USBMS 77 SCSI: Response LUN: 0x00 (Test Unit
Ready) (Good)
89 4.702043 host 3.8.1 USBMS 95 SCSI: Prevent/Allow Medium Removal LUN:
0x00 PREVENT
90 4.702069 3.8.1 host USB 64 URB_BULK out
91 4.702072 host 3.8.1 USB 64 URB_BULK in
92 4.703006 3.8.1 host USBMS 77 SCSI: Response LUN: 0x00 (Prevent/Allow
Medium Removal) (Good)

93 4.703052 host 3.8.1 USBMS 95 SCSI: Synchronize Cache(10) LUN: 0x00
(LBA: 0x00000000, Len: 0)
94 4.703066 3.8.1 host USB 64 URB_BULK out
95 4.703067 host 3.8.1 USB 64 URB_BULK in
96 4.704146 3.8.1 host USBMS 77 SCSI: Response LUN: 0x00 (Synchronize
Cache(10)) (Good)
97 4.704149 host 3.8.1 USBMS 95 SCSI: Request Sense LUN: 0x00
98 4.704177 3.8.1 host USB 64 URB_BULK out
99 4.704179 host 3.8.1 USB 64 URB_BULK in
100 4.705032 3.8.1 host USBMS 82 SCSI: Data In LUN: 0x00 (Request Sense
Response Data)
101 4.705035 host 3.8.1 USB 64 URB_BULK in
102 4.705053 3.8.1 host USBMS 77 SCSI: Response LUN: 0x00 (Request
Sense) (Good)
105 6.740272 host 3.8.1 USBMS 95 SCSI: Test Unit Ready LUN: 0x00
106 6.740323 3.8.1 host USB 64 URB_BULK out
107 6.740326 host 3.8.1 USB 64 URB_BULK in
108 6.740410 3.8.1 host USBMS 77 SCSI: Response LUN: 0x00 (Test Unit
Ready) (Good)
195 8.709417 host 3.8.1 USBMS 95 SCSI: Prevent/Allow Medium Removal LUN:
0x00 ALLOW
196 8.709441 3.8.1 host USB 64 URB_BULK out
197 8.709445 host 3.8.1 USB 64 URB_BULK in
198 8.709645 3.8.1 host USBMS 77 SCSI: Response LUN: 0x00 (Prevent/Allow
Medium Removal) (Good)

The patch indeed works. The kindle does not disconnect.
I also tried with a sync on a mounted filesystem. The effect is the
same: No disconnect.

Calling "eject /dev/sde" still works as it should.

Tested-by: Matthias Schwarzott <***@gentoo.org>

Regards
Matthias

[Hans de Goede]

Hi,

Matthias reports that the Amazon Kindle automatically removes its
emulated media if it doesn't receive another SCSI command within about
one second after a SYNCHRONIZE CACHE. It does so even when the host
has sent a PREVENT MEDIUM REMOVAL command. The reason for this
behavior isn't clear, although it's not hard to make some guesses.
At any rate, the results can be unexpected for anyone who tries to
access the Kindle in an unusual fashion, and in theory they can lead
to data loss (for example, if one file is closed and synchronized
while other files are still in the middle of being written).
To avoid such problems, this patch creates a new usb-storage quirks
flag telling the driver always to issue a REQUEST SENSE following a
SYNCHRONIZE CACHE command, and adds an unusual_devs entry for the
Kindle with the flag set. This is sufficient to prevent the Kindle
from doing its automatic unload, without interfering with proper
operation.
Another possible way to deal with this would be to increase the
frequency of TEST UNIT READY polling that the kernel normally carries
out for removable-media storage devices. However that would increase
the overall load on the system and it is not as reliable, because the
user can override the polling interval. Changing the driver's
behavior is safer and has minimal overhead.

Thanks, patch looks good to me:

Reviewed-by: Hans de Goede <***@redhat.com>

Regards,

Hans

---
[as1953]
drivers/usb/storage/transport.c | 7 +++++++
drivers/usb/storage/unusual_devs.h | 12 ++++++++++++
include/linux/usb_usual.h | 2 ++
3 files changed, 21 insertions(+)
Index: usb-devel/drivers/usb/storage/transport.c
===================================================================
--- usb-devel.orig/drivers/usb/storage/transport.c
+++ usb-devel/drivers/usb/storage/transport.c
@@ -656,6 +656,13 @@ void usb_stor_invoke_transport(struct sc
need_auto_sense = 1;
}
+ /* Some devices (Kindle) require another command after SYNC CACHE */
+ if ((us->fflags & US_FL_SENSE_AFTER_SYNC) &&
+ srb->cmnd[0] == SYNCHRONIZE_CACHE) {
+ usb_stor_dbg(us, "-- sense after SYNC CACHE\n");
+ need_auto_sense = 1;
+ }
+
/*
* If we have a failure, we're going to do a REQUEST_SENSE
* automatically. Note that we differentiate between a command
Index: usb-devel/drivers/usb/storage/unusual_devs.h
===================================================================
--- usb-devel.orig/drivers/usb/storage/unusual_devs.h
+++ usb-devel/drivers/usb/storage/unusual_devs.h
@@ -2212,6 +2212,18 @@ UNUSUAL_DEV( 0x1908, 0x3335, 0x0200, 0x0
US_FL_NO_READ_DISC_INFO ),
/*
+ * The Amazon Kindle treats SYNCHRONIZE CACHE as an indication that
+ * the host may be finished with it, and automatically ejects its
+ * emulated media unless it receives another command within one second.
+ */
+UNUSUAL_DEV( 0x1949, 0x0004, 0x0000, 0x9999,
+ "Amazon",
+ "Kindle",
+ USB_SC_DEVICE, USB_PR_DEVICE, NULL,
+ US_FL_SENSE_AFTER_SYNC ),
+
+/*
* This device morphes spontaneously into another device if the access
* pattern of Windows isn't followed. Thus writable media would be dirty
Index: usb-devel/include/linux/usb_usual.h
===================================================================
--- usb-devel.orig/include/linux/usb_usual.h
+++ usb-devel/include/linux/usb_usual.h
@@ -86,6 +86,8 @@
/* lies about caching, so always sync */ \
US_FLAG(NO_SAME, 0x40000000) \
/* Cannot handle WRITE_SAME */ \
+ US_FLAG(SENSE_AFTER_SYNC, 0x80000000) \
+ /* Do REQUEST_SENSE after SYNCHRONIZE_CACHE */ \
#define US_FLAG(name, value) US_FL_##name = value ,
enum { US_DO_ALL_FLAGS };

[Hans de Goede]

Hi,

Hi,

I think it would be mildly better, but not a whole lot.  Since the
Kindle describes itself as having removable media, the kernel normally
probes it periodically to make sure the media remains present.  The
default probing interval is 2 seconds.  Reducing it to 0.9 seconds
doesn't represent an exorbitant additional load IMO -- especially since
Kindles don't tend to spend huge amounts of time connected to computers.

Ah, I did not know that the default polling interval was that low(ish),
given that the default indeed is already that low, then changing it to
0.8 seconds would not be a big change.  And we probably have a lot of
lower hanging fruit for unnecessary wakeups then that.

So we need to make a decision: Should the patch be merged, or should we
punt the issue to userspace tools?
On the plus side, the patch is rather small and non-invasive (although
it does allocate the last remaining bit in the 32-bit fflags field).
There's also the advantage of sending the extra command only when it is
needed, as opposed to increasing the overall frequency of TUR polling.
Any opinions?

1. It will work out of the box.
2. It only sends one extra command when needed.
3. It makes the block-device not break if user-space adjusts the poll interval to higher values.

FWIW my vote goes to the in kernel fix too.

Regards,

Hans

[Ulrich Windl]

Hi,

I think it would be mildly better, but not a whole lot. Since the
Kindle describes itself as having removable media, the kernel normally
probes it periodically to make sure the media remains present. The
default probing interval is 2 seconds. Reducing it to 0.9 seconds
doesn't represent an exorbitant additional load IMO ‑‑ especially since
Kindles don't tend to spend huge amounts of time connected to computers.

Ah, I did not know that the default polling interval was that low(ish),
given that the default indeed is already that low, then changing it to
0.8 seconds would not be a big change. And we probably have a lot of
lower hanging fruit for unnecessary wakeups then that.

So we need to make a decision: Should the patch be merged, or should we
punt the issue to userspace tools?
On the plus side, the patch is rather small and non‑invasive (although
it does allocate the last remaining bit in the 32‑bit fflags field).
There's also the advantage of sending the extra command only when it is
needed, as opposed to increasing the overall frequency of TUR polling.
Any opinions?

1. It will work out of the box.

...once you have the right kernel

2. It only sends one extra command when needed.
3. It makes the block‑device not break if user‑space adjusts the poll
interval to higher values.
Matthias
_______________________________________________
systemd‑devel mailing list
https://lists.freedesktop.org/mailman/listinfo/systemd‑devel

[Matthias Schwarzott]

Thank you for this patch, yes if this works it would IMHO be
a much better solution then the udev rule.

Thank you for the patch.

I think it would be mildly better, but not a whole lot. Since the
Kindle describes itself as having removable media, the kernel normally
probes it periodically to make sure the media remains present. The
default probing interval is 2 seconds. Reducing it to 0.9 seconds
doesn't represent an exorbitant additional load IMO -- especially since
Kindles don't tend to spend huge amounts of time connected to computers.
If it's merely a question of where to change the polling interval from
the default (automatically in the kernel or by userspace), that also
doesn't seem like a terribly important choice. Certainly a udev rule or
hwdb entry is a lot easier to maintain than special-case code in the
kernel.

One question though, if this works to fix the undesired ejects,
will an actual eject (using e.g. the eject utility as say
"sudo eject /dev/sda") still be seen as an eject by the kindle
after this ?

It should be. Maybe Matthias will try it and let us know.

Just for reference a direct answer here:
Yes, eject keeps working as it should.

Because that is actually kind of important for everyone using their
Kindle with Calibre, breaking that would not be good.

I also tried to upload a book using calibre (without disabling any
workarounds in calibre).
The full process did work fine.

Regards
Matthias