[Bug 453579] Re: corruption of large files reported with linux 2.6.31-14.46 on ext4

[beomuex]

** Also affects: linux via
http://bugzilla.kernel.org/show_bug.cgi?id=14354
Importance: Unknown
Status: Unknown

[mabovo]

Sory if I hijack this bug but seems that there is something related if I am not totally wrong:
I am using 9.10 with Mac2,1, ext4 on sda3.
When trying to copy a DVD like Snow or Leopard.iso (aprox. 7.5 GB) into an external HD *fat32), Nautilus stop the process in the middle displaying an error message with the following "Error writing the file: File too big"

The same iso can be copied into another partition of my internal hd like
sda4 (NTFS/Windows7) without errors.

[mabovo]

Architecture: i386
AudioDevicesInUse:
USER PID ACCESS COMMAND
/dev/snd/controlC0: mabovo 2942 F.... pulseaudio
CRDA: Error: [Errno 2] No such file or directory
Card0.Amixer.info:
Card hw:0 'Intel'/'HDA Intel at 0x90440000 irq 22'
Mixer name : 'SigmaTel STAC9221 A1'
Components : 'HDA:83847680,106b2200,00103401'
Controls : 21
Simple ctrls : 13
DistroRelease: Ubuntu 9.10
HibernationDevice: RESUME=UUID=720ad8ab-7e45-4167-8bdf-93b289eb6e50
MachineType: Apple Inc. MacBook2,1
Package: linux (not installed)
ProcCmdLine: BOOT_IMAGE=/boot/vmlinuz-2.6.31-14-generic root=UUID=65607e5f-473e-416d-b35b-4012db3c6f36 ro quiet splash
ProcEnviron:
SHELL=/bin/bash
LANG=pt_BR.UTF-8
LANGUAGE=pt_BR.UTF-8
ProcVersionSignature: Ubuntu 2.6.31-14.48-generic
RelatedPackageVersions:
linux-backports-modules-2.6.31-14-generic N/A
linux-firmware 1.24
Uname: Linux 2.6.31-14-generic i686
UserGroups: adm admin cdrom dialout lpadmin netdev plugdev sambashare
dmi.bios.date: 06/27/07
dmi.bios.vendor: Apple Inc.
dmi.bios.version: MB21.88Z.00A5.B07.0706270922
dmi.board.asset.tag: Base Board Asset Tag
dmi.board.name: Mac-F4208CAA
dmi.board.vendor: Apple Inc.
dmi.board.version: PVT
dmi.chassis.asset.tag: Asset Tag
dmi.chassis.type: 10
dmi.chassis.vendor: Apple Inc.
dmi.chassis.version: Mac-F4208CAA
dmi.modalias: dmi:bvnAppleInc.:bvrMB21.88Z.00A5.B07.0706270922:bd06/27/07:svnAppleInc.:pnMacBook2,1:pvr1.0:rvnAppleInc.:rnMac-F4208CAA:rvrPVT:cvnAppleInc.:ct10:cvrMac-F4208CAA:
dmi.product.name: MacBook2,1
dmi.product.version: 1.0
dmi.sys.vendor: Apple Inc.

[mabovo]

** Attachment added: "AlsaDevices.txt"
http://launchpadlibrarian.net/34432272/AlsaDevices.txt

[mabovo]

** Attachment added: "AplayDevices.txt"
http://launchpadlibrarian.net/34432273/AplayDevices.txt

[mabovo]

** Attachment added: "ArecordDevices.txt"
http://launchpadlibrarian.net/34432274/ArecordDevices.txt

[mabovo]

** Attachment added: "BootDmesg.txt"
http://launchpadlibrarian.net/34432275/BootDmesg.txt

[mabovo]

** Attachment added: "Card0.Amixer.values.txt"
http://launchpadlibrarian.net/34432276/Card0.Amixer.values.txt

[mabovo]

** Attachment added: "Card0.Codecs.codec.0.txt"
http://launchpadlibrarian.net/34432277/Card0.Codecs.codec.0.txt

[mabovo]

** Attachment added: "CurrentDmesg.txt"
http://launchpadlibrarian.net/34432278/CurrentDmesg.txt

[mabovo]

** Attachment added: "IwConfig.txt"
http://launchpadlibrarian.net/34432279/IwConfig.txt

[mabovo]

** Attachment added: "Lspci.txt"
http://launchpadlibrarian.net/34432280/Lspci.txt

[mabovo]

** Attachment added: "PciMultimedia.txt"
http://launchpadlibrarian.net/34432283/PciMultimedia.txt

[mabovo]

** Attachment added: "ProcInterrupts.txt"
http://launchpadlibrarian.net/34432287/ProcInterrupts.txt

[mabovo]

** Attachment added: "RfKill.txt"
http://launchpadlibrarian.net/34432292/RfKill.txt

[mabovo]

** Attachment added: "Lsusb.txt"
http://launchpadlibrarian.net/34432281/Lsusb.txt

[mabovo]

** Attachment added: "ProcCpuinfo.txt"
http://launchpadlibrarian.net/34432284/ProcCpuinfo.txt

[mabovo]

** Attachment added: "ProcModules.txt"
http://launchpadlibrarian.net/34432291/ProcModules.txt

[mabovo]

** Attachment added: "UdevLog.txt"
http://launchpadlibrarian.net/34432295/UdevLog.txt

[mabovo]

** Attachment added: "XsessionErrors.txt"
http://launchpadlibrarian.net/34432300/XsessionErrors.txt

[mabovo]

** Attachment added: "UdevDb.txt"
http://launchpadlibrarian.net/34432293/UdevDb.txt

[mabovo]

** Attachment added: "WifiSyslog.txt"
http://launchpadlibrarian.net/34432298/WifiSyslog.txt

[Steve Langasek]

mabovo, your issue is completely unrelated.

[Steve Langasek]

** Changed in: linux (Ubuntu Karmic)
Milestone: ubuntu-9.10 => karmic-updates

[Manuel Bua]

Since it seems the fix has been planned for karmic-updates, should we expect ext3 to be used as the default fs when installing Karmic?
I'm quite worried about the impact this bug could have on new users migrating to Ubuntu.

[Steve Langasek]

Setting the milestone does not mean that a fix is planned for karmic-
updates. So far, it doesn't appear that Scott's original problem is
reproducible for anyone else. We will stay on this bug to try to
confirm it and find a fix, but we aren't going to change the default fs
for a bug that only one person is seeing.

[Steven Post]

@mabovo: a regular fat32 filesystem only supports files up to about 4GB,
wich explains your problem.

I haven't expirienced this on ext4 yet, but I did notice some corruption on ext3 a while back, nothing important, but it could have been corruptions introduced with the transfer of the file. I don't know how you downloaded it, but it might be a clue (http, ftp, bittorrent,..?), not every protocol has the same level of corruption checking.
Although I'm afraid it is in the filesystem, in my case with ext3 it was a torrent, first hash checking passed, a month later it didn't.

[ian]

You can add me as a second user seeing the problem. My original report
is here:

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/459839

I've seen the bug with two independent installations to ext4. In my
case, a fsck does seem to repair the problem, making a non-bootable
system bootable again. The occurrence of disk errors is sporadic.

You asked about RAID early in the thread. I have a raid controller on my
mobo, which is currently not being used. The SATA drive is plugged
directly into the main connections, and is reported as /dev/sda

I wouldn't mention this at all, but for the fact that some live CD
versions of linux (gnuparted LiveCD, for example) gave me fits when they
recognized the RAID controller, tried to associate the drive with the
RAID device, and therefore prevented me from reformatting the drive.
Made me wonder if there might be some quirky interplay deep in the
device stack leading to false positive RAID detections.

As reported in the original bug, I am running Kubuntu 9.10 rc 64-bit on
intel quad core machine and an intel x25-m ssd. (I don't think this is
one of the infamous intel SSD bugs because an alternative OS ran w/o
problems)

[Steve Langasek]

Post by ian
I've seen the bug with two independent installations to ext4. In my
case, a fsck does seem to repair the problem, making a non-bootable
system bootable again. The occurrence of disk errors is sporadic.

That doesn't sound at all like the bug Scott has described.

[Scott Kitterman]

Proposed release note:

There have been some reports of data corruption with fresh (not
upgraded) ext4 file systems with large files (over 512MB). The issue
is under investigation. Users who routinely manipulate large files may
want to consider using ext3 file systems until this issue is resolved.

[Neumarke]

On the issue how many people are seeing this problem, and I hope I'm not
misunderstanding the relationships between bugs here:

This bug is "assigned to" linux-kernel-bugs #14354 in which Linus Torvalds himself claims to be seeing filesystem corruption, starting here:
http://bugzilla.kernel.org/show_bug.cgi?id=14354#c117

Are these bugs related or not?

[Steve Langasek]

Neumarke,

The relation to that upstream bug is tenuous at best. The upstream bug:
- is reported against a newer kernel than the one we're shipping
- is reported to only happen when ext4 is on top of the DM layer, whereas Scott's case was ext4 on a raw device
- is reported in connection with an unclean shutdown and subsequent fsck, whereas Scott reported corruption of files without an unclean shutdown (but no mention in this bug of whether the corruption requires an intervening reboot/fsck to appear - Scott, please clarify)

So that upstream bug link should be dropped; it really doesn't look like
the same bug.

** Changed in: linux
Importance: Unknown => Undecided

** Changed in: linux
Status: Unknown => New

** Changed in: linux
Remote watch: Linux Kernel Bug Tracker #14354 => None

[Steve Langasek]

Documented at
<https://wiki.ubuntu.com/KarmicKoala/ReleaseNotes#Possible%20corruption%20of%20large%20files%20with%20ext4%20filesystem>:

There have been some reports of data corruption with fresh (not
upgraded) ext4 file systems using the Ubuntu 9.10 kernel when writing to
large files (over 512MB). The issue is under investigation, and if
confirmed will be resolved in a post-release update. Users who routinely
manipulate large files may want to consider using ext3 file systems
until this issue is resolved. (453579)


** Changed in: ubuntu-release-notes
Status: New => Fix Released

[Kai Blin]

Steve, I can confirm that in my setup.

Test is easy, as described by Scott.

I've copied over the first iso I found on my PC to my fileserver running
an ext4 /data partition. Then I had some fun with md5sum:

kai at woodstock:/data/iso$ md5sum en_win_xp_pro_n.iso
138468d380b84e6b9e9a8648efb97143 en_win_xp_pro_n.iso
kai at woodstock:/data/iso$ md5sum en_win_xp_pro_n.iso
d6b2bc09fc4df1876005f20b62364f6e en_win_xp_pro_n.iso
kai at woodstock:/data/iso$ sync
kai at woodstock:/data/iso$ md5sum en_win_xp_pro_n.iso
91cf62eee1e159774c8acee03cf88f39 en_win_xp_pro_n.iso
kai at woodstock:/data/iso$ md5sum en_win_xp_pro_n.iso
d11a45c61466f2b22757e0e449e2fe90 en_win_xp_pro_n.iso
kai at woodstock:/data/iso$ md5sum en_win_xp_pro_n.iso
f8e682be3590d48e0c266d15a804e3af en_win_xp_pro_n.iso
kai at woodstock:/data/iso$ md5sum en_win_xp_pro_n.iso
d11a45c61466f2b22757e0e449e2fe90 en_win_xp_pro_n.iso
kai at woodstock:/data/iso$ md5sum en_win_xp_pro_n.iso
d11a45c61466f2b22757e0e449e2fe90 en_win_xp_pro_n.iso

Note that it seems to stabilized on d11a45c61466f2b22757e0e449e2fe90
after bouncing off from there once. Also note that calling sync seems to
have no effect on the bug.

That's certainly a fun one.

[Kai Blin]

Oh, I forgot to mention that d11a45c61466f2b22757e0e449e2fe90 is not the
correct checksum the file is supposed to have.

[Bob McElrath]

I have seen problems like this with large files on multiple fs's and
ultimately it was a RAM problem. Scott, can you run memtester and/or
memtest86 at bootup to verify that you don't have bad RAM? Is your CPU
overclocked? CPU errors can also be detected with burn* programs
(cpuburn package). A rare RAM problem can cause bitflips that you
wouldn't notice except in large files.

[Scott James Remnant]

Post by Bob McElrath
I have seen problems like this with large files on multiple fs's and
ultimately it was a RAM problem. Scott, can you run memtester and/or
memtest86 at bootup to verify that you don't have bad RAM? Is your CPU
overclocked? CPU errors can also be detected with burn* programs
(cpuburn package). A rare RAM problem can cause bitflips that you
wouldn't notice except in large files.

Running memtest was one of the first things I did ;-) Likewise I
performed a read/write test of the drive and it was fine

Scott

[Lemmiwinks]

A few days ago, a video file in my home folder, which was over 300MB large, became unusable. Nautilus says the file has 0 bytes. When I try to open it, every player reports, that the stream does not contain any data.
Unfortunately I can not tell when exactly or what the file corruption caused.
When there are more reports like mine, I would suggest to withdraw karmic completely...

[Lemmiwinks]

Forgot to mention, that I've got actually an Ext3 file system, which I
updated to Ext4 soon after Jaunty was released, with no problems at all.

[Martin Pitt]

Post by Lemmiwinks
Forgot to mention, that I've got actually an Ext3 file system, which I
updated to Ext4 soon after Jaunty was released, with no problems at all.

Scott, did you also upgrade your's to ext4, or was that a clean
mkfs.ext4?

[Scott James Remnant]

Post by Martin Pitt

Post by Lemmiwinks
Forgot to mention, that I've got actually an Ext3 file system, which I
updated to Ext4 soon after Jaunty was released, with no problems at all.

Scott, did you also upgrade your's to ext4, or was that a clean
mkfs.ext4?

Clean ext4

Scott

[Martin Jackson]

I have an ext4 fs that I created in jaunty as a fresh ext4 fs (during
the jaunty beta cycle).

The fs is on lvm and is close to 1 TB in size...it's 92% full with mp4
files in frequent use, and I have not yet seen this issue.

I upgraded this machine to karmic just over a week ago.

[Mackenzie Morgan]

Lemmiwinks: Sounds more like the old 0-byte bug that was in Jaunty's
ext4. Scott's bug keeps the files the same size.

I'm getting this too with the .isos I downloaded today. Mine is not a
ext3 --> ext4 conversion. It was formatted as ext4 by the Karmic alpha 3
or 4 installer. Unlike Scott and Ian, I am not using an SSD.

Zsync'd iso:
098824768ee3d46dcb60e8cd1fe37f61 kubuntu-9.10-desktop-amd64.iso

Torrented iso:
290ef766fdef0bda13df5c3d7ba7c163 kubuntu-9.10-desktop-amd64.ipv6.iso

Should be:
5a996e0d794e35509d0275d411a3e737 *kubuntu-9.10-desktop-amd64.iso
according to http://nl.releases.ubuntu.com/releases/kubuntu/karmic/MD5SUMS

[Axos]

I just finished a clean install of 9.10 (new default
partitions...nothing retained from previous install) on a Toshiba
notebook with an old-school 120 GB parallel ATA drive (whatever you call
the drives that came before SATA) and 2 GB of RAM.

I ran the following commands:

openssl rand -out foo 629145600
md5sum foo
sync
md5sum foo
cp foo bar
md5sum foo bar
sync
md5sum foo bar
openssl rand -out foo2 1073741824
md5sum foo2
sync
md5sum foo2
cp foo2 bar2
md5sum foo2 bar2
md5sum foo bar foo2 bar2 > sums
# rebooted the system
md5sum foo bar foo2 bar2

All the sums were consistent. No variation. Either my system doesn't
have the problem -or- there is something else which triggers it. For
instance, maybe the files need to be some odd size rather than a clean
multiple of 1 MB. The sizes I used above were 600 * 1024 * 1024 and 1024
* 1024 * 1024. I'll retry the test with an additional 17 bytes added to
the file sizes to see if that makes any difference. I'll post again if
it does.

[David Warde-Farley]

@Kai Blin: Can you please confirm the kernel version this was happening
with?

[Kai Blin]

I'm seeing this on 2.6.31.4 of the beagleboard armel kernel from Launchpad. However, this might be a false alarm on my side, pointing at a hardware issue instead. I've reformatted the partition to ext3 and I'm still seeing similar effects. This is an external USB drive, which might be one part of the issue.
Sorry about the noise, this bug looked like a perfect match.

[beomuex]

** Changed in: linux
Importance: Undecided => Unknown

** Changed in: linux
Status: New => Unknown

** Changed in: linux
Remote watch: None => Linux Kernel Bug Tracker #14354

[beomuex]

** Changed in: linux
Importance: Unknown => Undecided

** Changed in: linux
Status: Unknown => New

** Changed in: linux
Remote watch: Linux Kernel Bug Tracker #14354 => None

** Changed in: linux
Importance: Undecided => Unknown

** Changed in: linux
Status: New => Unknown

** Changed in: linux
Remote watch: None => Linux Kernel Bug Tracker #14354

[nutznboltz]

http://lkml.org/lkml/2009/3/24/415

[Juan A. Romero]

I am confused about this bug. All comments speak about freshly-created
ext4 filesystems, as well as the Karmic Release Notes. But what about
already-present filesystems? Right now my / is ext3 and /home is ext4 on
Jaunty. If I do a dist-upgrade to Karmic, will I be affected? What about
converting / later from ext3 to ext4? And what if I install Karmic from
scratch, but leaving /home untouched?

[Mackenzie Morgan]

We don't know the cause, so that's hard to answer. So far it seems that ext3
--> ext4 conversions are safe. Kind of makes sense, since the on-disk system
is a bit different. As to whether created-by-Karmic or in-use-by-Karmic is
the trouble here, we don't know yet. I think only 3 people so far have hit
this, and we all were running unstable for development reasons, so we had
created-by-Karmic filesystems. It's going to take more people reproducing it
to find out if created-by-Jaunty-and-used-in-Karmic is problematic as well.

[fimbulvetr]

I tried to reproduce this on my Latitude D630 with an intel x25-m, 9.10
fresh format/install mounted raw, and was unable to.

Immediately after grabbing ubuntu-9.10-desktop-amd64.iso, the md5sum was
dc51c1d7e3e173dcab4e0b9ad2be2bbf, and did not change even after a
reboot.

[Starcraftmazter]

fimbulvetr - did you try editing the file?

On that note, I'm doing a fresh install of 9.10 with ext4 on my laptop
around the start of next week, and I'm wondering if anyone can suggest
some methods to try and reproduce the bug. So far I'm thinking about
obtaining a very large file, copying it around the HD, and modifying it.

I'm wondering if changing the file around the 536870912th byte would be
a useful thing to do?

[tellapu]

Thanks so much for working on this critical issue. I wait to install
Karmic till it is fixed, so please hurry up :-) As I often have large
files (around 1 GB).

[GonzO]

I think Steve was right at post #84: the link to the linux kernel bug
should be dropped, as all of the circumstances of this bug are different
from the one in the link. How did this upstream link get re-
established?

[Bug Watch Updater]

** Changed in: linux
Status: Unknown => Confirmed

[Axos]

OK, the bug is in kernel 2.6.32. Kosmic, er, Karmic Koala is 2.6.31. No
wonder I wasn't able to reproduce it.

http://bugzilla.kernel.org/show_bug.cgi?id=14354

[Steve Langasek]

** Changed in: linux
Importance: Unknown => Undecided

** Changed in: linux
Status: Confirmed => New

** Changed in: linux
Remote watch: Linux Kernel Bug Tracker #14354 => None

[aldebx]

@ Kai Blin,
it should be made clear that testing with an external USB drive is not at all a reliable test. I've got through _several_ USB drives that systematically corrupted large files regardless to the HardDisk, filesystem and host computer used. This happens especially with large capacity harddisks plugged into cheap usb controllers (although that also happened to me once with an average one).

@Starcraftmazter
since the MD5 sum as all other hashes are conceived to ensure files have not been tampered with or corrupted it would _definitely_ change if you edit them! The hash file (read MD5/SHA, etc) HAVE TO change after you edit the file! Otherwise you would have been so lucky to have found a weakness in the hash algorithm.

[Kai Blin]

@aldebx
Dunno, connecting all of my drives to all of my other boxes, I don't see any issues like that. However, I think I've already identified the system used as the real cause of my particular issue.

[Bryan Quigley]

Did everyone affected do a memtest like the suggestion earlier
(https://bugs.launchpad.net/ubuntu/+source/linux/+bug/453579/comments/88)

I have had the EXACT same symptoms ('cept it was a 4.7 GB ISO), on ext3
and it was a SINGLE bad line in memtest. So please run memtest Scott.
Or anyone else affected by the changing md5sums. Thanks!

(Why would it work on ext3 and not ext4? maybe because ext4 reads faster
and would be more likely to trip the bad memory)

[Starcraftmazter]

@aldebx

Of course I realise this, perhaps I need to elaborate my idea. I mean,
since the error apparently occurs when large files are edited, a test
should be devised whereby changes are made to a large file, saved, and
then un-done and saved - and the before and after checksums compared, to
see if there in fact is a problem with writing large files.

Furthermore, since the problem allegedly happens around bits at the
512MB mark, so my idea is to write a program to take an X number of
blocks before and after this bit, and swap them. X must be even to
ensure every block is swapped with another. I am thinking of swapping
1000 blocks before the point with 1000 points after the point. Using
fsync and running the program twice should ensure that both changes are
written, and the second undoes the first - thus if two hashes of the
file are taken, one before and one after the experiment, they will
either be identical if no problems occured or different if there is in
fact a problem.

So my question is, would this be a good test to do? I will probably have
time to do it tomorrow.

[unggnu]

I also had this issue but I can't really nail it down. The explanation
that it has only something to do with Kernel 2.6.32-rc* makes sense. I
used it several times on my two systems. I got different md5 for the
same file and if I played a video which was affected the player stops
and the hard disks runs all the time and the systems hangs for a minute
or more. It looks like a part was missing and the driver was searching
for it.

I even shot down my whole testing system. I have a small testing partition to test Linux outside the VM. After some xorg edgers installation and restart I got an fsck problem which asks for confirmation, some inode problem. After confirmation it asks again several times so I run it with -y which delete a lot of files. Afterwards the system didn't boot anymore.
I have installed grub on the partition instead of the MBR, maybe that resulted under some special circumstances in this problem but I have done this since a long time and never had problem with it.

So actually using of 2.6.32 makes more sense since I use ext4 since
quite some time. I haven't had a problem since sticking with the default
Kernel but this problems doesn't just pop up. You don't realize that a
huge file has changed until actually checking/using the whole thing.

[Matt]

This might be a stupid question, but Karmic does ship with the Kernel
2.6.31-14.48 and not 14.46 right? Can anybody elaborate?

I did a fresh install with newly created ext4 partitions and have not
yet encountered anything. Well I didnt really try to produce an error
since this is my production machine. I have md5sumed entire folders with
Gigs of data after moving them ... no errors.

[Ramon]

Well I am not sure but heres a copy of what mine said after a clean
install "Linux 2.6.31-14-generic-pae" ...also I have not really notice
any corruption except couple hours ago when i tried to reinstall grub2 i
got a "segmentation fault" error.

[Starcraftmazter]

Hello.

I have wrote a C program to implement the test I described above. Currently, it checks 100x 8K blocks around the 512MB mark, by swapping them with each other, back to front. Running the program twice should thus result in an identical file. Using this program, you can check that read/write ability of your filesystem, in particular around with 512MB mark in a file (both before, at, and after).
http://codepad.org/msApnHFY
If you don't have gcc installed (it isn't by default):
sudo apt-get install build-essential
To compile:
gcc -o tester tester.c


Further, I wrote a perl script to simplify testing:
http://codepad.org/djmLaJ0l
To run:
chmod +x tester.pl
./tester.pl

If those links disappear at some stage, the programs can be found here:
http://starcraftmazter.net/launchpad/453579/


Both me and my friend have ran the test on the Ubuntu iso itself. I am
using a 64bit install of 9.10 and he is using a 32bit install of 9.10.
The kernel used for our tests is the default 2.6.31-14-generic. We are
both on ext4.

Both of our tests came up fine, and read/write works perfectly and the
before/after hashes are the same, hence we could not observe any
problem.


I would encourage anyone experiencing this problem to run the above tests and see what happens, in an effort to isolate the problem.


Cheers

[black]

I have a freshly installed ubuntu 9.10 with 2 newly created ext4
partitions (45GiB for / and 870GiB for /home). I did not encounter any
problem so far. (The /home drive contains ~500GiB of films) Also the
tester program of comment #112 doesn't reveal any problems. So I'm lucky
for now and will report if the problem emerges.

[Starcraftmazter]

Furthermore, I should state the full kernel version that the final
version of Ubuntu (which we did our testing on) is 2.6.31-14.48 and not
14-46. Is there a fix from 46 to 48?

[Steve Langasek]

Post by Starcraftmazter
Furthermore, I should state the full kernel version that the final
version of Ubuntu (which we did our testing on) is 2.6.31-14.48 and not
14-46. Is there a fix from 46 to 48?

No. The bug title reflects the version of the kernel on which the error was
first seen.
--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
slangasek at ubuntu.com vorlon at debian.org

[Starcraftmazter]

@Steve Langasek
I understand that, though what I'm wondering is whether there was any change from 46 to 48, which could have fixed this issue.

[Mackenzie Morgan]

I had -14.48 when I hit it.

[grof]

I have a fresh installation of Karmic Koala, and I've already have two times corrupted fs.
Ubuntu does not boot and complain about fs it cannot mount.
I have to do fsck in order to repair the things.

But the perl script above (of Starcraftmazter) said that hashes are
equal.

[muadnem]

Is this not fixed via
https://bugs.edge.launchpad.net/ubuntu/+source/linux/+bug/317781 ?

If so, will the iso images for Karmic be updated anytime soon, or will
this only be available post-install?


Maybe off base, sorry if so.

[muadnem]

I guess it would help if I pasted the right link... Please ignore the
previous link.

http://bugzilla.kernel.org/show_bug.cgi?id=14354

[muadnem]

Here is my reasoning..

"One change that we did make between 2.6.31 and 2.6.32 is that we enable
journal checksums by default."

"by default" suggests that a 2.6.31 could be built with journal
checksums enabled?

And maybe I'm reading wrong but it doesn't look DM specific..

[Starcraftmazter]

Hmmm are the two issues (corrupted fs and corrupted large files)
related?

[Steve Langasek]

Post by Starcraftmazter
Hmmm are the two issues (corrupted fs and corrupted large files)
related?

No.
--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
slangasek at ubuntu.com vorlon at debian.org

[muadnem]

Woops. I missed the 'corruption is not detected by fsck', part. Seems
like, with the elusive nature of this bug, everyone should be reporting
their memtest and fsck status.

[Steve Langasek]

Post by muadnem
Woops. I missed the 'corruption is not detected by fsck', part. Seems
like, with the elusive nature of this bug, everyone should be reporting
their memtest and fsck status.

In general, people who aren't actually seeing the bug described here should
not be reporting anything. All that does is make the bug log harder to
extract information from.
--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
slangasek at ubuntu.com vorlon at debian.org

[Leonardo Montecchi]

Is this bug related somewhat to the following:

http://bugzilla.kernel.org/show_bug.cgi?id=14385
http://marc.info/?l=linux-kernel&m=125504643703877&w=4

?

** Bug watch added: Linux Kernel Bug Tracker #14385
http://bugzilla.kernel.org/show_bug.cgi?id=14385

[p2k]

I too would recommend a full memtest to anyone encountering data
corruption.

[unggnu]

I could confirm it again with the default Ubuntu Kernel. I was downloading a compilation of files with Bittorrent while the battery went out. There was no problem afterwards and the files seem to be downloaded fine later on but I got errors. So I started a rehash of the compilation and it found around 40 defective chunks which needed to be redownloaded. Afterwards the file check works fine so I guess the problem might have something to do with crashes/blackout in combination with ext4.
There is no problem that some recently saved data is gone after a crash but at least it should be recognized through the journal and marked as such.

[Brian Rogers]

The BitTorrent crash scenario doesn't indicate a bug. The only way for
an application to know about uncommitted writes is to scan the file (for
example by rehashing in this instance). To avoid doing this every time
it's started, it saves a record of what parts have been downloaded. In a
crash, this record may be more up to date than what's actually saved to
the disk.

[Bryan Quigley]

For bittorrent (I'm assuming using transmision) check out this bug:
https://bugs.launchpad.net/ubuntu/+source/transmission/+bug/445592

[Ramon]

I restarted my system the hard way and it would not boot. So i checked it
from another install with fsck and it found some errors. I'm beginning to
worry about a possible future corruption. What is the status of this
problem?

[Mackenzie Morgan]

Ramon:
That's not what this bug is about. Improperly rebooting runs the risk
of breaking your system on *any* filesystem. This bug is about
*individual files* which are very large becoming corrupt and NOT
having any effect on fsck.

[Ramon]

yea thats true. Have you experience any more corruption since?

On Sat, Nov 7, 2009 at 3:20 PM, Mackenzie Morgan <macoafi at gmail.com>

Post by Mackenzie Morgan
That's not what this bug is about. Improperly rebooting runs the risk
of breaking your system on *any* filesystem. This bug is about
*individual files* which are very large becoming corrupt and NOT
having any effect on fsck.
--
corruption of large files reported with linux 2.6.31-14.46 on ext4
https://bugs.launchpad.net/bugs/453579
You received this bug notification because you are a direct subscriber
of the bug.
Status in The Linux Kernel: New
Status in Ubuntu Release Notes: Fix Released
Status in ?linux? package in Ubuntu: Triaged
Status in ?linux? source package in Karmic: Triaged
There are worrying reports of filesystem corruption on ext4 in karmic.
12:36 < Keybuk> this whole ext4 thing is worrying me
12:36 < Keybuk> I just downloaded an iso image, md5sum didn't match
12:36 < Keybuk> downloaded it into an ext3 partition, matched just fine
12:59 < Keybuk> and I know mvo has seen bugs with corrupted .debs in
/var/cache/apt/archives
12:59 < Keybuk> which seems to imply its any file large enough to use lots
of extents
I'm opening this bug report so that this bug gets tracked & triaged for
karmic. If we're unable to isolate the issue, we should consider rolling
back to ext3 as the default filesystem in the installer.
ProblemType: Bug
Architecture: amd64
**** List of CAPTURE Hardware Devices ****
card 0: Intel [HDA Intel], device 0: AD198x Analog [AD198x Analog]
Subdevices: 1/1
Subdevice #0: subdevice #0
USER PID ACCESS COMMAND
/dev/snd/controlC0: vorlon 3350 F.... pulseaudio
Card hw:0 'Intel'/'HDA Intel at 0xee240000 irq 17'
Mixer name : 'Analog Devices AD1981'
Components : 'HDA:11d41981,17aa2025,00100200'
Controls : 20
Simple ctrls : 11
Date: Fri Oct 16 16:01:26 2009
DistroRelease: Ubuntu 9.10
HibernationDevice: RESUME=UUID=f108133c-6b9d-4d28-9058-0b3a0c5549b4
MachineType: LENOVO 6371CTO
Package: linux-image-2.6.31-14-generic 2.6.31-14.46
no product info available
no card
ProcCmdLine: root=/dev/mapper/hostname-root ro
PATH=(custom, user)
LANG=en_US.UTF-8
SHELL=/bin/bash
ProcVersionSignature: Ubuntu 2.6.31-13.44-generic
RelatedPackageVersions: linux-firmware 1.22
SourcePackage: linux
Uname: Linux 2.6.31-13-generic x86_64
dmi.bios.date: 12/27/2006
dmi.bios.vendor: LENOVO
dmi.bios.version: 7IET23WW (1.04 )
dmi.board.name: 6371CTO
dmi.board.vendor: LENOVO
dmi.board.version: Not Available
dmi.chassis.asset.tag: No Asset Information
dmi.chassis.type: 10
dmi.chassis.vendor: LENOVO
dmi.chassis.version: Not Available
dmi.product.name: 6371CTO
dmi.product.version: ThinkPad T60
dmi.sys.vendor: LENOVO

--
??G??
--
http://www.goeazy.com
http://twitter.com/ram130
http://www.myspace.com/everythinglegit

[Mackenzie Morgan]

I haven't downloaded anymore large files since then on the basis that it'd be a
waste of bandwidth

Post by Ramon
yea thats true. Have you experience any more corruption since?
On Sat, Nov 7, 2009 at 3:20 PM, Mackenzie Morgan <macoafi at gmail.com>

Post by Mackenzie Morgan
That's not what this bug is about. Improperly rebooting runs the risk
of breaking your system on *any* filesystem. This bug is about
*individual files* which are very large becoming corrupt and NOT
having any effect on fsck.
--
corruption of large files reported with linux 2.6.31-14.46 on ext4
https://bugs.launchpad.net/bugs/453579
You received this bug notification because you are a direct subscriber
of the bug.
Status in The Linux Kernel: New
Status in Ubuntu Release Notes: Fix Released
Status in ?linux? package in Ubuntu: Triaged
Status in ?linux? source package in Karmic: Triaged
There are worrying reports of filesystem corruption on ext4 in karmic.
12:36 < Keybuk> this whole ext4 thing is worrying me
12:36 < Keybuk> I just downloaded an iso image, md5sum didn't match
12:36 < Keybuk> downloaded it into an ext3 partition, matched just fine
12:59 < Keybuk> and I know mvo has seen bugs with corrupted .debs in
/var/cache/apt/archives
12:59 < Keybuk> which seems to imply its any file large enough to use
lots of extents
I'm opening this bug report so that this bug gets tracked & triaged for
karmic. If we're unable to isolate the issue, we should consider rolling
back to ext3 as the default filesystem in the installer.
ProblemType: Bug
Architecture: amd64
**** List of CAPTURE Hardware Devices ****
card 0: Intel [HDA Intel], device 0: AD198x Analog [AD198x Analog]
Subdevices: 1/1
Subdevice #0: subdevice #0
USER PID ACCESS COMMAND
/dev/snd/controlC0: vorlon 3350 F.... pulseaudio
Card hw:0 'Intel'/'HDA Intel at 0xee240000 irq 17'
Mixer name : 'Analog Devices AD1981'
Components : 'HDA:11d41981,17aa2025,00100200'
Controls : 20
Simple ctrls : 11
Date: Fri Oct 16 16:01:26 2009
DistroRelease: Ubuntu 9.10
HibernationDevice: RESUME=UUID=f108133c-6b9d-4d28-9058-0b3a0c5549b4
MachineType: LENOVO 6371CTO
Package: linux-image-2.6.31-14-generic 2.6.31-14.46
no product info available
no card
ProcCmdLine: root=/dev/mapper/hostname-root ro
PATH=(custom, user)
LANG=en_US.UTF-8
SHELL=/bin/bash
ProcVersionSignature: Ubuntu 2.6.31-13.44-generic
RelatedPackageVersions: linux-firmware 1.22
SourcePackage: linux
Uname: Linux 2.6.31-13-generic x86_64
dmi.bios.date: 12/27/2006
dmi.bios.vendor: LENOVO
dmi.bios.version: 7IET23WW (1.04 )
dmi.board.name: 6371CTO
dmi.board.vendor: LENOVO
dmi.board.version: Not Available
dmi.chassis.asset.tag: No Asset Information
dmi.chassis.type: 10
dmi.chassis.vendor: LENOVO
dmi.chassis.version: Not Available
dmi:bvnLENOVO:bvr7IET23WW(1.04):bd12/27/2006:svnLENOVO:pn6371CTO:pvrThink
dmi.product.name: 6371CTO
dmi.product.version: ThinkPad T60
dmi.sys.vendor: LENOVO

--
Mackenzie Morgan
http://ubuntulinuxtipstricks.blogspot.com
apt-get moo

[Pete Graner]

Looks like the is mostly solved upstream by reverting patch
d0646f7b636d067d715fab52a2ba9c6f0f46b0d7 and adding the patch
487caeef9fc08c0565e082c40a8aaf58dad92bbb.

@apw could you or csurbhi build a test kernel and post here so folks can
test?

Thanks

~pete

** Changed in: linux
Importance: Undecided => Unknown

** Changed in: linux
Status: New => Unknown

** Changed in: linux
Remote watch: None => Linux Kernel Bug Tracker #14354

[Steve Langasek]

No, this is not upstream bug #14354. There is no overlap between the
described problems.

** Changed in: linux
Importance: Unknown => Undecided

** Changed in: linux
Status: Unknown => New

** Changed in: linux
Remote watch: Linux Kernel Bug Tracker #14354 => None

** Summary changed:

- corruption of large files reported with linux 2.6.31-14.46 on ext4
+ in-place corruption of large files *without fsck or reboot* reported with linux 2.6.31-14.46 on ext4

[Piscium]

I am running Karmic 9.10 on a old Pentium 4 computer with PATA drives. I
copied a 2.6 GByte file from an ext4 partition to another ext4
partition, then to a ext2, then to the original ext4. No problem. All
files have the same md5.

[unggnu]

"Improperly rebooting runs the risk of breaking your system on *any* filesystem."
Sorry, but this is not true imho. I have never had a similar problem with ext3. Yes, there are some file systems like XFS which just deletes the data of a whole file if the computer crashes or is restarted but I guess the goal should be especially with a journaling file system to prevent errors like this. Not to mention that the size of the file is zero with XFS so you don't assume that everything is fine.
Like I said there is no problem that after a hard reboot some shortly changed data is lost but this has to be diagnosed and dealt with through the file system so that no corrupt files are saved without even realizing it.

[The Loeki]

I see there's a lot of discussion going on about this bug, I'll just
drop in my 5c:

I cleanly installed/updated a x64 Karmic w/ext4 filesystems on my MacBook 5,1 (no other OSes installed).
Due to the nature of my work I downloaded over a dozen ISO's of different HTTP/FTPs, none of which failed their MD5 sums.

Yesterday, I cleanly installed/updated a x32 Karmic on some measly
Centrino/Pentium M 1,6 Ghz laptop and copied, amongst others, 7 DVD
ISO's to it from an external NTFS harddrive, with no apparent issue or
corruption (though I'd have to check the MD5's to make absolutely sure).

So for me, this issue doesn't seem to exist. Then again, both
filesystems are clean-and-simple 3 primary partition layouts (10GB
/,RAM*1~1,5 swap, remainder /home, handmade during install), no hw/sw
RAID/LVM or whatever.

[Andrew M.]

I think I actually *have* seen this...

My setup:
Fileserver, 1.5TB array on JFS bunch of big files (videos mostly, some ISO's and the like - but none of these files *ever* change)
one client with 400GB internal SATA HDD on ext4, running Karmic with 2.6.31-15-generic AMD64 kernel
one client with 1.5TB external USB HDD on ext4, running Karmic with 2.6.31-15-generic i386 kernel

I invoke rsync from the clients as
rsync -axvc root at server:/home/some/dir /home/backup-dir

on *both* clients, i had to run rsync about 3 times until i no longer
saw changes.

that is to say that there existed differences in the files even during
2nd rsync, which simply shouldn't be.

Also worth noting that another Karmic machine with ext3 and a ppc kernel
doesn't see this problem. Ran rsync once, and then the second pass
didn't change any files.


There doesn't seem to be any rhyme or reason to what files were corrupt or wrong on the clients after the first sync, they weren't the same between the two clients and there didn't seem to be any correlation between increased size and frequency or anything. The "giantest" files (8 GB's or so) transferred correctly the first time.

I've still got this elaborate test set up in place, and I'm *very*VERY*
keen to get this worked out so that I can move to Karmic on the server
(no way in hell I'm upgrading until this gets sorted out!!!)

Please let me know if any more specific tests would help or anything.

[Andrew M.]

more info from my setup:
i have done memtest on all boxes, everything is fine
network is a wired network (not thinking it should matter, SSH would barf if packets were coming in with errors,)
the smallest file for which i saw corruption was about 120MB,
the incidence seemed to be about 6 corrupt files every {delete everything on client, reboot client, run rsync, run rsync again to see what checksums were wrong} iteration, during which minimally 400GB and about 10,000 files were transferred

[Andrew M.]

oh yes, one more thing,
all fs's were created & formatted by the Karmic installer, using the release media

[Ramon]

Wow now that's a test!!! I think karmic corrupted my windows7 and two data
partitions. I installed karmic on a brand new 21days old 500GB hard drive.
Been transfering files for 2 weeks from my failing 320GB. After that was
done I tried booting back into windows7, failed. Karmic crashes occasionally
for no reason!! Decided to run start up repair, no problems then I ran
chkdsk, all of corrupted files on each partition! ..to top it off disk
utility is reporting my hard drive now has a bad sector!! Sumone help me
before I go insane

On Nov 11, 2009 9:26 PM, "Andrew M." <2bitoperations at gmail.com> wrote:

oh yes, one more thing,
all fs's were created & formatted by the Karmic installer, using the release
media

-- in-place corruption of large files *without fsck or reboot* reported with
linux 2.6.31-14.46 on...

[Mackenzie Morgan]

Er...this is only for ext4. Win7 does not run on ext4. Sounds like that bad
sector is to blame. Just because it's new doesn't mean it's not broken.

[Ramon]

I used Karmic to create the partitions, 2 NTFS, SWAP & an EXT4. The bad
sector didn't show up until I was doing copying files...so far it says One
bad sector. It just seems ironic this corruption problem is here then this
happens.

On Thu, Nov 12, 2009 at 1:39 AM, Mackenzie Morgan <macoafi at gmail.com>

Post by Mackenzie Morgan
Er...this is only for ext4. Win7 does not run on ext4. Sounds like that bad
sector is to blame. Just because it's new doesn't mean it's not broken.
--
in-place corruption of large files *without fsck or reboot* reported with
linux 2.6.31-14.46 on ext4
https://bugs.launchpad.net/bugs/453579
You received this bug notification because you are a direct subscriber
of the bug.
Status in The Linux Kernel: New
Status in Ubuntu Release Notes: Fix Released
Status in ?linux? package in Ubuntu: Triaged
Status in ?linux? source package in Karmic: Triaged
There are worrying reports of filesystem corruption on ext4 in karmic.
12:36 < Keybuk> this whole ext4 thing is worrying me
12:36 < Keybuk> I just downloaded an iso image, md5sum didn't match
12:36 < Keybuk> downloaded it into an ext3 partition, matched just fine
12:59 < Keybuk> and I know mvo has seen bugs with corrupted .debs in
/var/cache/apt/archives
12:59 < Keybuk> which seems to imply its any file large enough to use lots
of extents
I'm opening this bug report so that this bug gets tracked & triaged for
karmic. If we're unable to isolate the issue, we should consider rolling
back to ext3 as the default filesystem in the installer.
ProblemType: Bug
Architecture: amd64
**** List of CAPTURE Hardware Devices ****
card 0: Intel [HDA Intel], device 0: AD198x Analog [AD198x Analog]
Subdevices: 1/1
Subdevice #0: subdevice #0
USER PID ACCESS COMMAND
/dev/snd/controlC0: vorlon 3350 F.... pulseaudio
Card hw:0 'Intel'/'HDA Intel at 0xee240000 irq 17'
Mixer name : 'Analog Devices AD1981'
Components : 'HDA:11d41981,17aa2025,00100200'
Controls : 20
Simple ctrls : 11
Date: Fri Oct 16 16:01:26 2009
DistroRelease: Ubuntu 9.10
HibernationDevice: RESUME=UUID=f108133c-6b9d-4d28-9058-0b3a0c5549b4
MachineType: LENOVO 6371CTO
Package: linux-image-2.6.31-14-generic 2.6.31-14.46
no product info available
no card
ProcCmdLine: root=/dev/mapper/hostname-root ro
PATH=(custom, user)
LANG=en_US.UTF-8
SHELL=/bin/bash
ProcVersionSignature: Ubuntu 2.6.31-13.44-generic
RelatedPackageVersions: linux-firmware 1.22
SourcePackage: linux
Uname: Linux 2.6.31-13-generic x86_64
dmi.bios.date: 12/27/2006
dmi.bios.vendor: LENOVO
dmi.bios.version: 7IET23WW (1.04 )
dmi.board.name: 6371CTO
dmi.board.vendor: LENOVO
dmi.board.version: Not Available
dmi.chassis.asset.tag: No Asset Information
dmi.chassis.type: 10
dmi.chassis.vendor: LENOVO
dmi.chassis.version: Not Available
dmi.product.name: 6371CTO
dmi.product.version: ThinkPad T60
dmi.sys.vendor: LENOVO

--
??G??
--
http://www.goeazy.com
http://twitter.com/ram130
http://www.myspace.com/everythinglegit

[unggnu]

@Ramon
If you use Karmic you can check your whole hard disk with SMART. Check the bad sector count after an extended test. If there are some it is more likely that this was the cause then Karmic.

I guess this should be done by everyone who have problems and have
already run memtest.