Discussion:
Issue 59315 in chromium: Hangs, then crashes in Wikimedia Commons (displaying an images gallery in the multilingual "Nature" category)
c***@googlecode.com
2010-10-15 05:34:52 UTC
Permalink
Status: Unconfirmed
Owner: ----
Labels: Type-Bug Pri-2 Area-Undefined

New issue 59315 by verdyp: Hangs, then crashes in Wikimedia Commons
(displaying an images gallery in the multilingual "Nature" category)
http://code.google.com/p/chromium/issues/detail?id=59315

Chrome Version : 8.0.552.0 dev
URLs (if applicable) :

http://commons.wikimedia.org/wiki/Category:Nature

Other browsers tested:
Add OK or FAIL after other browsers where you have tested this issue:
Safari 4: OK
Firefox 3.x: OK
IE 7: OK
IE 8: OK

What steps will reproduce the problem?
1. Open URL.

What is the expected result?


What happens instead?

Page never displays, and the browser hangs using 100% CPU (all Windows
versions, bug persistant in LOTS of Wikimedia sites, since Chrome v4 Beta
more than 2 years ago)

Please provide any additional information below. Attach a screenshot if
possible.

Once again, a resource downloader never finishes (even if the page is
already fully loaded in cache). This seems to occur each time Chrome
performs mny HTTP requests just to validate its cache, for external scripts
or image, or AJAX requests.

The bug occurs randomly on almost all Wikimedia sites, whatever their
contents, but much more frequently if there are many PNG images, or if
there are many translations needing many fonts (one for each script).

This seems related to the PNG renderer (when it performs progressive
display of images, possibly only images that were compressed using the
Adam7 progressive display).

Every time, this causes a CPU core to use 100% of its time, and system
tracing reveals an infinite loop of worker process creation attempts. This
heavily loads the system, and if we leave it running without closing the
tab very fast, it will crash the system by making it fully unresponsive (we
can't even start the Windows Task Manager that never finishes its own
display refresh, before its own display is cleared and restarted again.)

It's still completely impossible to create a crash dump (not even with
windbg) because all these repetitive processes created are terminating
normally and never crash : they just exits and a parent processs continues
to trying to recreate this process continuously.

Process creation in Windows is the most expensive system call. There is
absolutely no end to this tight process creation loop (which may be
generated by the Chrome crash handler trying to resolve the unexpected
process exit by retrying it.

In windbg, we just see that all processes are idle, but the process id
counters from Windows are growing at a very fast rate, and loops back
continuously. All those dying processes are also consuming lots of system
ressources (Windows clean ups processes only in a background garbage
collection loop, which is not fast enough to preserve ressources, including
those many file handles, management of shared memory blocks with costly TLB
lookup table updates...).

In Windbg, you can break this tight loop, but all you see is a list of
processes that performs absolutely nothing else than just dying... I can't
even generate a stack trace of where the loop is caused (so this is
probably within a custom exception handler, specifically built with some
unusual frame format that is invisible from usual stack frames that Windbg
can detect and report.)

Apparently the parent process is not receiving some system event that it
should receive early when the procss starts (in some DLLEntry point?), and
performs a "TerminateThread" to force the child process to exit cleanly.

However, this URL (the Nature category on Wikimedia Commons) is showing
this bug occuring always (it's completely impossible to show this category
in Chrome). Wikimedia Commons is one of the most seriously affected sites
with this bug occuring extremely frequently, just after major Wikipedia
editions (such as English or French) due to their increased use of
javascripts, PNG image illustrations or galleries, and AJAX plugins.

Apparently, you are queing several processes for loading external
ressources from the browser cache, but not all od them can fit the event
queues in the parent process, but one of these processes terminates before
the parent process can receive the event that the child process could not
send back to the parent. The child then exists prematurely, and the parent
process retries it. This sems to occur when some small threshold count of
processes in the creation queue is exceeded.

This is a VERY SERIOUS bug signaled many times, and never corrected since
Chrome v4 Beta around March 2008 (4 major releases, since more than TWO
YEARS, and very soon a 5th one with v8 showing the same bug), and it is
generally compeltely unpredictable. he bug is so serious that we must
constantly take care of forcing closing the browser window very fast before
it crashes the whole system (or causes a CPU overheat on notebooks that
will power it down abruptly). Eahc time we experiment data losses if we
can't close the hanged tab very fast (don't wait more than a dozen of
seconds with the SPINNING ICON, in a fast clockwise rotation.
--
Automated mail from issue updates at http://crbug.com/
Subscription options: http://groups.google.com/a/chromium.org/group/chromium-bugs
c***@googlecode.com
2010-11-22 18:39:14 UTC
Permalink
Updates:
Labels: -Area-Undefined Area-Internals

Comment #2 on issue 59315 by ***@chromium.org: Hangs, then crashes
in Wikimedia Commons (displaying an images gallery in the
multilingual "Nature" category)
http://code.google.com/p/chromium/issues/detail?id=59315

Can you provide the requested information please?
--
Automated mail from issue updates at http://crbug.com/
Subscription options: http://groups.google.com/a/chromium.org/group/chromium-bugs
c***@googlecode.com
2010-12-01 08:19:07 UTC
Permalink
Comment #7 on issue 59315 by verdyp: Hangs, then crashes in Wikimedia
Commons (displaying an images gallery in the multilingual "Nature" category)
http://code.google.com/p/chromium/issues/detail?id=59315

I don't know how you test it, I can do all tests I want, on ALL my
machines, in clean installations of Windows, either as a host OS, or in a
VM, ALL of them result in the same hang (the only thing I get is first a
white screen without any content, and the loading icon with fast clockwise
spinning, which progressively becomes slower and then very slow (and a
constantly increasing resource usage in memory), until I get the alert that
Chrome does not reply : I must absolutely reply "Yes" to the question
asking if I want to cancel the page loading, because if I replay "No" and
let the loading continue, it will completely hang the whole system (forcing
to reboot).

This reproduces ALWAYS on ALL my installations of Windows and independantly
of the type of processor (Intel or AMD, single or double core) or graphics
card (ATI or nVidia), or machine type (HP notebook, ASUS notebook, OEM
desktops), and as well on various versions of Windows (XP, Vista or Seven),
and with distinct antivirus kits (it also happens as well when I don't
install any antivirus and just use the default Windows Defender with the
default recommanded settings in the Windows Security Center), either 32-bit
or 64-bit, with 1GB or with 8 GB of memory.

The only thing in common is that this is a **French** retail version of
Windows (with all their standard system updates applied), with **French**
version of Chrome and of the antivirus tool.

There are also NO addon or plugins except the deault builtin plugins
preinstalled within Chrome itself. The tested page anyway does NOT include
or require any one of these plugins. It is curious that you still don't
have any incoming crash report from my PC when I click "Yes" to cancel the
page loading and get the failed tab content filled in blue.

Once again, I REPEAT : how can I create such crash dump that would exhibit
this behavior ? I WANT such debugging option, otherwise you will constnatly
say that this bug is not reproducible. But most probablty you have NEVER
tried to use any FRENCH retail version of Windows and Chrome
simultaneously, becaue this bug would be obvious to you as it is REALLY
EASY to reproduce IMMEDIATELY on ALL the systems I could access to by just
typing this URL in the address bar
"http://commons.wikimedia.org/wiki/Category:Nature".
--
Automated mail from issue updates at http://crbug.com/
Subscription options: http://groups.google.com/a/chromium.org/group/chromium-bugs
c***@googlecode.com
2010-12-01 09:03:41 UTC
Permalink
Comment #8 on issue 59315 by verdyp: Hangs, then crashes in Wikimedia
Commons (displaying an images gallery in the multilingual "Nature" category)
http://code.google.com/p/chromium/issues/detail?id=59315

Note: my geolocalisation performed by Google on my IP is EXTREMELY slow to
reply to the request, this may be an issue, because there's no reason why
Wikimedia performs such geolocalisation, and this must be done by the
browser itself.

(I see a LOT of REPEATED geolocalisation requests performed with the help
of a Google server, when looking at a HTTP log of these requests by
installing a proxy and instructing first Chrome to connect to the Internet
by this proxy; the proxy itself is fully transparent and is not the cause
of the problem, becuase the bug occurs as wrll without using any such
proxy : I just used this proxy to try investigating what was really
happening in the background and that Chrome refuses to let me see).

Once again, this bug never occurs on the same page with any other browser
than Chrome (Safari, IE8, IE9 Beta, Opera, Firefox) which DO NOT exhibit
any one of these geolocalisation requests that Chrome is apparently
constantly trying to do ITSELF in the background and without informing
users.

Could it be caused by my ISP ? It is the second largest ISP in France (SFR,
group Vivendi-Universal), but I have no other problems with websites
performing their own geolocalisations. Could this be an issue in your
implementation of Google geolocalisation ? Or by an incorrect analysis of
what it returns from my (dynamic) IP ? Anyway, in mid-2008 I was not using
SFR, but Orange (the first ISP in France) and I had exactly the same issue
on pages of Wikimedia sites displaying PNG image thumbnails. I have
reported this hanging bug many times, and it's been uite al ong time
without any working solution found or without any reply or suggestion on
how to investigate this issue or without any workaround.

PLEASE PROVIDE ME WITH HINTS about how to investigate the issue. I
absolutely have no idea how to give you more details. (And please don't
repeat that this may be caused by a rootkit or virus or malware, I have
never been able to detect any one since many years on my systems, using all
the possible tools, and I know how to create a clean installation of a
retail version of Windows in a VM : it also happens when using a prebuilt
VM downloaded from Microsoft and booting from it in a Windows Server
installation, where this same bug also occurs).
--
Automated mail from issue updates at http://crbug.com/
Subscription options: http://groups.google.com/a/chromium.org/group/chromium-bugs
c***@googlecode.com
2010-12-02 18:56:47 UTC
Permalink
Comment #9 on issue 59315 by ***@chromium.org: Hangs, then crashes in
Wikimedia Commons (displaying an images gallery in the
multilingual "Nature" category)
http://code.google.com/p/chromium/issues/detail?id=59315

I don't see any hang or crash on Google Chrome 8.0.552.215 (Official Build
67652) and 9.0.597.0 (Official Build 67679). I'm on English OS though.

From one of your comment, you said that you are seeing a blue page. GPU
sometimes causes this. But this was happening even before GPU was
introduced. Anyways, I would suggest you to disable GPU and see if it helps
with latest versions.

I'm not sure what's the difference between French OS and English OS. You
can actually change your OS to English and check if it works.

Geolocation issue might be a different one. I would suggest you to file a
separate bug for this.

If you see a hang tab, use instructions @
http://www.chromium.org/for-testers/bug-reporting-guidelines/hanging-tabs

If you see a crash, use instructions @
http://www.chromium.org/for-testers/bug-reporting-guidelines/reporting-crash-bug
--
Automated mail from issue updates at http://crbug.com/
Subscription options: http://groups.google.com/a/chromium.org/group/chromium-bugs
c***@googlecode.com
2010-12-02 20:34:55 UTC
Permalink
Comment #10 on issue 59315 by verdyp: Hangs, then crashes in Wikimedia
Commons (displaying an images gallery in the multilingual "Nature" category)
http://code.google.com/p/chromium/issues/detail?id=59315

No I have DEFINTELY NOT said this was the Windows BSOD. The system
immediately won't crash (it will crash if I let it continue without
canceling the page loading, because it will hang rapidly the whole system
with 100% CPU usage). ***REREAD***.

I see the blue **TAB** (within the browser, after the browser hanged).
There's no GPU issue (and REMEMBER: this happens on ALL the PC's that I
have tried, in various versions of Windows and with various hardwares, even
the simplest). This also happens in a VM installation of Windows: only the
tabs of windows within this VM are affected when the hang occurs.

This most often causes other random tabs to also become blue, even though
there was nothing active in them and if they were displaying a page from
another domain (for example the default Google.com search page, or a
Wikisource page).

This leads to me to the conclusion that this is a renderer plugin issue,
because plugin subprocesses are shared between tabs for the same plugin.

I've tried to disable ALL plugins (from the about:plugins page), and this
still happens. So this must be within an internal renderer built within
Chrome, or directly within its interprocess mechanism (most probably the
builtin PNG image renderer).

It is EXTREMELY easy to reproduce in that Category:Nature page of Commons
where the bug occurs ALSWAYS and immediately, without being able to display
anything from that page. The bug seems related to the presence of many
external PNG images on the same page as external javascript resources,
above some threshold, and to the presence of international characters from
several system fonts (Geolocalisation performed by Chrome also adds other
background resources to be requested by the browser).

Once again, no problem in any other browser. It may be something in the
interprocess synchronisation when accessing to the browser cache from
multiple rendering processes simultaneously). When I looked at what was
hapenning on the netweok (by using an external procy), I just noticed that
the geolocalisation requests were the last performed one just before the
hang. I don't know exactly when this geolocalisation was introduced, but
this bug is there since very long now (since eaarly betas of Chrome v4 more
than 2 years ago).

And NONE of your instructions are working for hanging tabs. It's simply
impossible to generate any dump from these failing tabs (the UI becomes
very rapidly not responsive enough to be able perform the dump request, and
it's impossible to generate a meaningful dump after the blue tab has been
displayed, because the hanging processes have already been killed).

Sometimes I will even see a "chrome.exe" process remaining permanently in
memory in a stale status, fully invisible, even after I have closed Chrome
completely in all windows (I'll have to use the Windows Process Manager to
kill them as there remains no UI for them). Apparently, they are remaining
there because they are holding a system lock.

This happens on all types of display (including virtual displays such as
RDP from remote Windows sessions in terminal server, where there's
absolutely NO involded GPU).

Your suggestion of disabling the GPU is also simply impossible to perform.
There's no such option available (even Windows, instructed to use its core
VGA driver, or running in "failsafe mode" with network support will still
use some hardware to display somthing. I can't run such tests without any
display, except through a VM where the brower still hangs exactly with the
same way).

Really, all seems to demonstrate that this is an interprocess
synchronization issue.
--
Automated mail from issue updates at http://crbug.com/
Subscription options: http://groups.google.com/a/chromium.org/group/chromium-bugs
c***@googlecode.com
2010-12-02 22:49:02 UTC
Permalink
Comment #17 on issue 59315 by verdyp: Hangs, then crashes in Wikimedia
Commons (displaying an images gallery in the multilingual "Nature" category)
http://code.google.com/p/chromium/issues/detail?id=59315

Where could I send a private email containing a private link to another
server where I could upload this dump file ? Is there another way to use
these dump files and extract the info you may want ?
--
Automated mail from issue updates at http://crbug.com/
Subscription options: http://groups.google.com/a/chromium.org/group/chromium-bugs
c***@googlecode.com
2010-12-02 23:22:22 UTC
Permalink
Comment #18 on issue 59315 by ***@chromium.org: Hangs, then crashes in
Wikimedia Commons (displaying an images gallery in the
multilingual "Nature" category)
http://code.google.com/p/chromium/issues/detail?id=59315

Dump files should not be that big. How are you generating those files?

"Disable GPU" meant "disable gpu acceleration in Chrome", not "turn off
your GPU (magically)" That is, pass --disable-accelerated-compositing to
Chrome.

I think the 3 people who have starred this bug are: you, me, and Sunand. =)
If somebody else has this problem, they would've spoken up already.
--
Automated mail from issue updates at http://crbug.com/
Subscription options: http://groups.google.com/a/chromium.org/group/chromium-bugs
c***@googlecode.com
2010-12-02 23:30:42 UTC
Permalink
Comment #19 on issue 59315 by verdyp: Hangs, then crashes in Wikimedia
Commons (displaying an images gallery in the multilingual "Nature" category)
http://code.google.com/p/chromium/issues/detail?id=59315

I am generating it from the Windows 7 Task Manager: right click on the
chrome.exe process, then select "create dump file"; I see the alert box
showing the URL to chrome.dmp.

It is REALLY that big !
--
Automated mail from issue updates at http://crbug.com/
Subscription options: http://groups.google.com/a/chromium.org/group/chromium-bugs
c***@googlecode.com
2010-12-02 23:42:55 UTC
Permalink
Comment #20 on issue 59315 by verdyp: Hangs, then crashes in Wikimedia
Commons (displaying an images gallery in the multilingual "Nature" category)
http://code.google.com/p/chromium/issues/detail?id=59315

Same issue when running "chrome --disable-accelerated-compositing" to open
that URL !
--
Automated mail from issue updates at http://crbug.com/
Subscription options: http://groups.google.com/a/chromium.org/group/chromium-bugs
c***@googlecode.com
2010-12-02 23:51:05 UTC
Permalink
Comment #21 on issue 59315 by verdyp: Hangs, then crashes in Wikimedia
Commons (displaying an images gallery in the multilingual "Nature" category)
http://code.google.com/p/chromium/issues/detail?id=59315

Anyway, uisng windbg to load the dump file here is what I see :


0:000> K100
Child-SP RetAddr Call Site
00000000`0022eb98 00000000`740d2d92 wow64cpu!CpupSyscallStub+0x9
00000000`0022eba0 00000000`7414d07e wow64cpu!Thunk0Arg+0x5
00000000`0022ec60 00000000`7414c549 wow64!Wow64SystemServiceEx+0x1ce
00000000`0022ecb0 00000000`77ac82c8 wow64!Wow64LdrpInitialize+0x429
00000000`0022f200 00000000`77ac7423 ntdll!RtlResetRtlTranslations+0x1b08
00000000`0022f700 00000000`77ab2e2e ntdll!RtlResetRtlTranslations+0xc63
00000000`0022f770 00000000`00000000 ntdll!LdrInitializeThunk+0xe

There are 4 threads (Windbg takes a considerable time to parse the threads
& processes status, requiring about 3 minutes to analyze it)
--
Automated mail from issue updates at http://crbug.com/
Subscription options: http://groups.google.com/a/chromium.org/group/chromium-bugs
c***@googlecode.com
2010-12-02 23:55:08 UTC
Permalink
Comment #22 on issue 59315 by ***@chromium.org: Hangs, then crashes in
Wikimedia Commons (displaying an images gallery in the
multilingual "Nature" category)
http://code.google.com/p/chromium/issues/detail?id=59315

You might be creating a full dump rather than a mini dump.
http://support.microsoft.com/kb/931673

You can email me a link to a .dmp file if you'd like.
--
Automated mail from issue updates at http://crbug.com/
Subscription options: http://groups.google.com/a/chromium.org/group/chromium-bugs
c***@googlecode.com
2010-12-03 00:31:44 UTC
Permalink
Comment #27 on issue 59315 by ***@chromium.org: Hangs, then crashes in
Wikimedia Commons (displaying an images gallery in the
multilingual "Nature" category)
http://code.google.com/p/chromium/issues/detail?id=59315

I downloaded the file and I'll try to take a look.
--
Automated mail from issue updates at http://crbug.com/
Subscription options: http://groups.google.com/a/chromium.org/group/chromium-bugs
c***@googlecode.com
2010-12-03 04:43:20 UTC
Permalink
Comment #28 on issue 59315 by verdyp: Hangs, then crashes in Wikimedia
Commons (displaying an images gallery in the multilingual "Nature" category)
http://code.google.com/p/chromium/issues/detail?id=59315

file removed
--
Automated mail from issue updates at http://crbug.com/
Subscription options: http://groups.google.com/a/chromium.org/group/chromium-bugs
c***@googlecode.com
2010-12-03 10:05:43 UTC
Permalink
Comment #29 on issue 59315 by verdyp: Hangs, then crashes in Wikimedia
Commons (displaying an images gallery in the multilingual "Nature" category)
http://code.google.com/p/chromium/issues/detail?id=59315

The link above no longer works, I have removed it, if someone else in the
developers need this file, please ask. I will also post a similar dump in
x86 pure 32 bit mode (generated from a new installation of Windows 7
(32-bit French) over Oracle VirtualBox (no GPU at all), that I am currently
installing (with all the default Windows Updates).

I have the same bug in another existing 64-bit installation of Windows 7
(which also has the same bug), but to make sure it will be clean from the
default settings of Windows, this news installation will ease the debugging
if you don't have to debug through the tricky WOW64 thunkings. I will once
again install the current Chrome v8 release on it.
--
Automated mail from issue updates at http://crbug.com/
Subscription options: http://groups.google.com/a/chromium.org/group/chromium-bugs
c***@googlecode.com
2010-12-03 10:45:21 UTC
Permalink
Comment #30 on issue 59315 by verdyp: Hangs, then crashes in Wikimedia
Commons (displaying an images gallery in the multilingual "Nature" category)
http://code.google.com/p/chromium/issues/detail?id=59315

I just redicovered an old backup of the Chrome v4.1 installer (downloaded
in last April).
With Chome 4.1 on exactly the SAME systems (including in a Oracle VM
Virtual Box installation of Windows 7 or Windows XP, just configured to use
a single processor core and a modest 1.5GB RAM configuration), after
uninstalling Chrome v7 or v8, no such hang occurs on this Wikimedia Commons
page ! The bug is systematic with Chrome v7 or v8.
--
Automated mail from issue updates at http://crbug.com/
Subscription options: http://groups.google.com/a/chromium.org/group/chromium-bugs
c***@googlecode.com
2010-12-03 11:13:34 UTC
Permalink
Comment #31 on issue 59315 by verdyp: Hangs, then crashes in Wikimedia
Commons (displaying an images gallery in the multilingual "Nature" category)
http://code.google.com/p/chromium/issues/detail?id=59315

This time, the following dump is generated with the current release
8.0.552.215 of Chrome, generated on a fresh of Windows 7 (32-bit), with all
default Windows Updates installed, in a "Oracle VM VirtualBox 3.2.10"
installation (configured to use only a single core, 1.5GB RAM, and no GPU
emulation for Direct3D, but the default Direct2D emulation driver as
installed by the "VirtualBox additional drivers CD for Guest OS'es").

http://net.rodage.com/img/chrome(2).dmp.zip
(35682 KB compressed, 95981 KB uncompressed)

(similar zize for the dump file, it is a bit higher because I have left the
tab hanging for longer and allowed Chrome to wait for longer time ; in the
Windows Process Manager, I see that this process never stops consuming more
memory, at a rate of about 8KB/s after it has resolved all DNS entries, and
finished downloading the page and several resources ; there is exactly the
same symptom with the clockwise spinning icon, and no content displayed,
just a white screen).

In that same VM (I made a backup of the VM disk image after reinstalling it
completely before installing each browser version of it, to make sure that
each browser will install and start in the same conditions), Chrome v4.1
has absolutely no problem to display this wikimedia page. And there is no
problem in IE7, IE8, Firefox, Opera 10, and Safari 4.1.

If then I use "Menu>About Chrome" from Chrome 4.1, it proposes the update
to Chrome 8, this installs successfully without any warning, and the page
no longer displays anything and hangs the same way, just like with a fresh
install of the current Chrome v8 release downladed from the Chrome website.
--
Automated mail from issue updates at http://crbug.com/
Subscription options: http://groups.google.com/a/chromium.org/group/chromium-bugs
c***@googlecode.com
2010-12-03 12:46:39 UTC
Permalink
Comment #32 on issue 59315 by verdyp: Hangs, then crashes in Wikimedia
Commons (displaying an images gallery in the multilingual "Nature" category)
http://code.google.com/p/chromium/issues/detail?id=59315

Here is what I get from windbg with this dump:



Microsoft (R) Windows Debugger Version 6.12.0002.633 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.


Loading Dump File [C:\Users\Philippe\AppData\Local\Temp\chrome (2).DMP]
User Mini Dump File with Full Memory: Only application data is available

Symbol search path is: C:\Symbols
Executable search path is:
Windows 7 Version 7600 MP (2 procs) Free x64
Product: WinNt, suite: SingleUserTS
Machine Name:
Debug session time: Fri Dec 3 09:20:57.000 2010 (UTC + 1:00)
System Uptime: 8 days 12:04:30.704
Process Uptime: 0 days 0:00:56.000
.........................WARNING: version overlaps wtsapi32
.....................
*** ERROR: Symbol file could not be found. Defaulted to export symbols for
ntdll.dll -
wow64cpu!CpupSyscallStub+0x9:
00000000`740d2dd9 c3 ret
0:000> .symfix +c:\symbols
0:000> ~0s
wow64cpu!CpupSyscallStub+0x9:
00000000`740d2dd9 c3 ret
0:000> kp
Child-SP RetAddr Call Site
00000000`001fea18 00000000`740d2d92 wow64cpu!CpupSyscallStub+0x9
00000000`001fea20 00000000`7414d07e wow64cpu!Thunk0Arg+0x5
00000000`001feae0 00000000`7414c549 wow64!RunCpuSimulation+0xa
00000000`001feb30 00000000`77ac82c8 wow64!Wow64LdrpInitialize+0x429
00000000`001ff080 00000000`77ac7423 ntdll!RtlResetRtlTranslations+0x1b08
00000000`001ff580 00000000`77ab2e2e ntdll!RtlResetRtlTranslations+0xc63
00000000`001ff5f0 00000000`00000000 ntdll!LdrInitializeThunk+0xe
0:000> lm
start end module name
00000000`00ef0000 00000000`00fe8000 chrome (deferred)
00000000`6b130000 00000000`6c6e4000 chrome_6b130000 (deferred)
00000000`6c810000 00000000`6cae0000 avcodec_52 (deferred)
00000000`6cae0000 00000000`6ced8000 pdf (deferred)
00000000`6d220000 00000000`6dca9000 icudt42 (deferred)
00000000`71c50000 00000000`71c87000 avformat_52 (deferred)
00000000`72280000 00000000`7229e000 userenv (deferred)
00000000`729a0000 00000000`729c3000 avutil_50 (deferred)
00000000`73280000 00000000`732ad000 ntmarta (deferred)
00000000`733a0000 00000000`7343e000 riched20 (deferred)
00000000`73580000 00000000`7371e000 comctl32 (deferred)
00000000`73720000 00000000`73776000 uxtheme (deferred)
00000000`73b30000 00000000`73b3f000 profapi (deferred)
00000000`73f40000 00000000`73f51000 wtsapi32 (no symbols)
00000000`73f50000 00000000`73f5c000 version (deferred)
00000000`740d0000 00000000`740d8000 wow64cpu (pdb symbols)
c:\symbols\wow64cpu.pdb\E9FF81183A9441CF90F25081E9C39A931\wow64cpu.pdb
00000000`740e0000 00000000`7413c000 wow64win (deferred)
00000000`74140000 00000000`7417f000 wow64 (pdb symbols)
C:\Program Files\Debugging Tools for Windows
(x64)\sym\wow64.pdb\612501691DDF4C66A9B98F6111F8B3EF1\wow64.pdb
00000000`74b30000 00000000`74b84000 oleacc (deferred)
00000000`74cd0000 00000000`74cdb000 secur32 (deferred)
00000000`75590000 00000000`755cb000 winmm (deferred)
00000000`757c0000 00000000`757cc000 CRYPTBASE (deferred)
00000000`757d0000 00000000`75830000 sspicli (deferred)
00000000`75830000 00000000`76479000 shell32 (deferred)
00000000`76510000 00000000`76567000 shlwapi (deferred)
00000000`76570000 00000000`7660d000 usp10 (deferred)
00000000`76610000 00000000`76615000 psapi (deferred)
00000000`76720000 00000000`7672a000 lpk (deferred)
00000000`768d0000 00000000`76970000 advapi32 (deferred)
00000000`76970000 00000000`76a70000 kernel32 (deferred)
00000000`76a70000 00000000`76a89000 sechost (deferred)
00000000`76db0000 00000000`76f0c000 ole32 (deferred)
00000000`76f10000 00000000`76f16000 nsi (deferred)
00000000`76f30000 00000000`76f65000 ws2_32 (deferred)
00000000`76f70000 00000000`76fb5000 Wldap32 (deferred)
00000000`76fc0000 00000000`77050000 gdi32 (deferred)
00000000`77050000 00000000`77140000 rpcrt4 (deferred)
00000000`77170000 00000000`771b6000 KERNELBASE (deferred)
00000000`771c0000 00000000`7724f000 oleaut32 (deferred)
00000000`77350000 00000000`773fc000 msvcrt (deferred)
00000000`77400000 00000000`7742e000 imm32 (deferred)
00000000`77490000 00000000`7755c000 msctf (deferred)
00000000`77560000 00000000`77660000 user32 (deferred)
00000000`77a80000 00000000`77c2b000 ntdll (export symbols)
ntdll.dll
00000000`77c60000 00000000`77de0000 ntdll_77c60000 (deferred)
0:000> !load wow64exts
0:000> !sw
Switched to 32bit mode
0:000:x86> kp
*** ERROR: Symbol file could not be found. Defaulted to export symbols for
chrome.dll -
ChildEBP RetAddr
0044c034 6b7e2f64 user32!NtUserGetDC+0x15
WARNING: Stack unwind information not available. Following frames may be
wrong.
0044c060 6b7df95d
chrome_6b130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x569731
0044c090 6b7db2e6
chrome_6b130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x56612a
0044dc54 6b7c39b1
chrome_6b130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x561ab3
0044dc84 6b7befe7
chrome_6b130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x54a17e
0044dca8 6b69a8bd
chrome_6b130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x5457b4
0044dd18 6b69b191
chrome_6b130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x42108a
0044e004 6b75b3a5
chrome_6b130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x42195e
0044e064 6b759215
chrome_6b130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x4e1b72
0044e07c 6b6de225
chrome_6b130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x4df9e2
0044e100 6b6bcc92
chrome_6b130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x4649f2
0044e11c 6b6bd05a
chrome_6b130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x44345f
0044e154 6b6bc8d6
chrome_6b130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x443827
0044e194 6b6bcdf0
chrome_6b130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x4430a3
0044e1e8 6b6bcc9d
chrome_6b130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x4435bd
0044e204 6b6bd05a
chrome_6b130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x44346a
0044e23c 6b6bc8d6
chrome_6b130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x443827
0044e27c 6b6a92cb
chrome_6b130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x4430a3
0044e2ac 6b6abc5e
chrome_6b130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x42fa98
0044e2d8 6b6abecd
chrome_6b130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x43242b
0044e34c 6b6abb53
chrome_6b130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x43269a
0044e36c 6b6187dc
chrome_6b130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x432320
0044ea0c 6b6186d1
chrome_6b130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x39efa9
0044ea2c 6b6bcdf0
chrome_6b130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x39ee9e
0044ea80 6b6bcc9d
chrome_6b130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x4435bd
0044ea9c 6b6bd05a
chrome_6b130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x44346a
0044ead4 6b6bc8d6
chrome_6b130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x443827
0044eb14 6b6bcdf0
chrome_6b130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x4430a3
0044eb68 6b6bcc9d
chrome_6b130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x4435bd
0044eb84 6b6bd05a
chrome_6b130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x44346a
0044ebbc 6b6bc8d6
chrome_6b130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x443827
0044ebfc 6b690f12
chrome_6b130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x4430a3
0044edac 6b69112b
chrome_6b130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x4176df
0044edd8 6b691018
chrome_6b130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x4178f8
0044ef9c 6b69112b
chrome_6b130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x4177e5
0044efc8 6b691018
chrome_6b130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x4178f8
0044f18c 6b6907d5
chrome_6b130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x4177e5
0044f1d8 6b6417f8
chrome_6b130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x416fa2
0044f210 6b7aec40
chrome_6b130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x3c7fc5
0044f278 6b90050a
chrome_6b130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x53540d
0044f2c8 6b90058a
chrome_6b130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x686cd7
0044f364 6b8fa64d
chrome_6b130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x686d57
0044f3a8 6b2606fa
chrome_6b130000!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x680e1a
0044f414 6b260b47 chrome_6b130000!DelayedLowerToken+0x124d71
0044f568 6b26080c chrome_6b130000!DelayedLowerToken+0x1251be
0044f590 6b1f5e72 chrome_6b130000!DelayedLowerToken+0x124e83
0044f5b0 6b1f5ef9 chrome_6b130000!DelayedLowerToken+0xba4e9
0044f5f0 6b20a8dd chrome_6b130000!DelayedLowerToken+0xba570
0044f61c 6b1f5c18 chrome_6b130000!DelayedLowerToken+0xcef54
0044f658 6b220fc2 chrome_6b130000!DelayedLowerToken+0xba28f
0044f8b0 6b133e95 chrome_6b130000!DelayedLowerToken+0xe5639
*** ERROR: Symbol file could not be found. Defaulted to export symbols for
chrome.exe -
0044fa54 00ef3c67 chrome_6b130000!ChromeMain+0x5b7
0044fae0 00f3817e chrome!SetNumberOfViews+0x1fa9
0044fae8 00f3be16 chrome!SetNumberOfViews+0x464c0
0044fb18 00ef4086 chrome!SetNumberOfViews+0x4a158
0044fb54 00f39e80 chrome!SetNumberOfViews+0x23c8
0044fb78 00f3807c chrome!SetNumberOfViews+0x481c2
0044fc08 76983677 chrome!SetNumberOfViews+0x463be
0044fc14 77c99d42 kernel32!BaseThreadInitThunk+0xe
0044fc54 77c99d15 ntdll_77c60000!__RtlUserThreadStart+0x70
0044fc6c 00000000 ntdll_77c60000!_RtlUserThreadStart+0x1b
--
Automated mail from issue updates at http://crbug.com/
Subscription options: http://groups.google.com/a/chromium.org/group/chromium-bugs
c***@googlecode.com
2010-12-03 13:39:02 UTC
Permalink
Comment #34 on issue 59315 by verdyp: Hangs, then crashes in Wikimedia
Commons (displaying an images gallery in the multilingual "Nature" category)
http://code.google.com/p/chromium/issues/detail?id=59315

Note: my antivirus is Avira Antivir 10.0 for Windows 7. Not sure that this
is the real problem, because I had the same problem with past versions 8
and 9 of Avira Antivir, and I still have the same hang with AVG Antivirus.
So this may be caused by Microsoft's updates (including those for Windows
Defender).
--
Automated mail from issue updates at http://crbug.com/
Subscription options: http://groups.google.com/a/chromium.org/group/chromium-bugs
c***@googlecode.com
2010-12-03 13:43:05 UTC
Permalink
Comment #33 on issue 59315 by verdyp: Hangs, then crashes in Wikimedia
Commons (displaying an images gallery in the multilingual "Nature" category)
http://code.google.com/p/chromium/issues/detail?id=59315

It may happen that this is effectively a security issue, because the
NtGetUserDc() in NTDLL is now restricted by Windows, and it has been hooked
by some rootkits.

Possibly, this is my antivirus that denies this call, because Chrome does
not own the security token for using this API.

I've searched on the Internet and found this article which may be
interesting (however I cannot confirm this, this goes to far behind my
level of competence in the windows API kernel security):

http://sec4vn.wordpress.com/2009/05/20/microsoft-windows-7-rc-7100-090421-gdi-driver-dos-kernel-vulnerabilty-win32k-sys/
--
Automated mail from issue updates at http://crbug.com/
Subscription options: http://groups.google.com/a/chromium.org/group/chromium-bugs
c***@googlecode.com
2010-12-03 13:51:07 UTC
Permalink
Comment #35 on issue 59315 by verdyp: Hangs, then crashes in Wikimedia
Commons (displaying an images gallery in the multilingual "Nature" category)
http://code.google.com/p/chromium/issues/detail?id=59315

Another discussion about this concerns the Norton 360 security suite. I
think this may in fact affect lots of antivirus kits, trying to protect us
from rootkits trying to hook these calls (so the security suites are
installing their own hooks on these APIs): they seem to think that Chrome
is acting like a rootkit and they block the call from being delivered to
the underlying OS API.
--
Automated mail from issue updates at http://crbug.com/
Subscription options: http://groups.google.com/a/chromium.org/group/chromium-bugs
c***@googlecode.com
2010-12-03 13:55:08 UTC
Permalink
Comment #36 on issue 59315 by verdyp: Hangs, then crashes in Wikimedia
Commons (displaying an images gallery in the multilingual "Nature" category)
http://code.google.com/p/chromium/issues/detail?id=59315

See also about "RelaunchChromeBrowserWithNewCommandLineIfNeeded":
- (closed?) Bug 47211,
- (closed?) Bug 43247 (related to builtin Hunspell's password manager),

and many others that have been posted about this Chrome function since May
4, 2010. Note that My old installer for Chrome 4.1 (which works), was
doanloaded in April 2010 and had no problem displaying this Wikimedia
Commons page.
--
Automated mail from issue updates at http://crbug.com/
Subscription options: http://groups.google.com/a/chromium.org/group/chromium-bugs
c***@googlecode.com
2010-12-03 14:23:14 UTC
Permalink
Comment #37 on issue 59315 by verdyp: Hangs, then crashes in Wikimedia
Commons (displaying an images gallery in the multilingual "Nature" category)
http://code.google.com/p/chromium/issues/detail?id=59315

Another possibility is that some code similar to the following attempts to
acquire an exclusive lock on the Desktop DC in one of the Chrome processes
(possibly to get some properties from it) :

Sub ConvertPixelsToTwips(X As Long, Y As Long)
Dim hDC As Long, hWnd As Long, RetVal As Long
Dim XPIXELSPERINCH As Long, YPIXELSPERINCH As Long

'' Retrieve the current number of pixels per inch, which is
'' resolution-dependent.
hDC = GetDC(0)
XPIXELSPERINCH = GetDeviceCaps(hDC, LOGPIXELSX)
YPIXELSPERINCH = GetDeviceCaps(hDC, LOGPIXELSY)
RetVal = ReleaseDC(0, hDC)

'' Compute and return the measurements in twips.
X = (X / XPIXELSPERINCH) * TWIPSPERINCH
Y = (Y / YPIXELSPERINCH) * TWIPSPERINCH
End Sub

And then this creates an inter-process mutual lock, because this Wikimedia
page needs to use many parallel renderers for the various PNG image
thumbnails. The autorecovery mechanism when User::GetDc(0) fails is to wait
for some fixed time and start again, indefinitely, in a very tight loop.

Generally, when a interprocess mutual interlocking issue occurs, before
retrying, the failing process should wait for a randomized time so that
other processes will not compete to the same exclusive ressource at the
same time.

Apparently, Chrome attempts to get the exclusive lock on the Desktop area
DC (and this may explain why this also makes very difficult to use other
processes, including the Windows Explorer desktop and system bar, and their
UI is severely affected). Do you really need this desktop DC ? Shouldn't
Chrome serialize and minimize the number of uses of this API between its
various processes, notably in their initialization when navigating to a new
page in a TAB that is already opened ?
--
Automated mail from issue updates at http://crbug.com/
Subscription options: http://groups.google.com/a/chromium.org/group/chromium-bugs
c***@googlecode.com
2010-12-03 16:08:02 UTC
Permalink
Comment #38 on issue 59315 by verdyp: Hangs, then crashes in Wikimedia
Commons (displaying an images gallery in the multilingual "Nature" category)
http://code.google.com/p/chromium/issues/detail?id=59315

When looking further in the sources around April 4 (the date when I
downloaded Chrome v4.1 which worked), it seems that there was extensive
works at this time to integrate a new IPC mechanism between renderer
windowed processes and the main UI process.
However, there was also the integration of the PNG image renderer as a
builtin (statically linked) renderer which is now windowless. Some old code
from the previous windowed PNG renderer (in a separate DLL) may have
survived which is incorrectly grabbing some IPC communication channel
(through desktop DC atoms ?) already used in another thread.
It is also significant that the other threads in the dump are all in a
status waiting for some Async IO channel to complete. The PNG rendering
thread seems not being able to get the desktop DC because it is already
hold in another thread, and all the rendering threads already hold it. One
of them will release the desktop DC and the other threads will continue
running as if they were still owning the user desktop/window DC (which can
be owned only in one thread per process). Then they fail for example to
read the internal process communication channel (I don't know how the IPC
channel is implemented and how it works as well to communicate within the
same process between internal threads), and all these renderers are dying
and are being restarted infinitely by some recovery mechanism. during their
initialization, they are all attempting to get properties from the desktop
(such as the TWIPS resolution, and color depth), and the desktop window
being constantly grabbed by many internal rendering threads in the main UI
process could be the cause of the severe UI degradation in all applications
(including in the Windows Explorer system bar or for running any other
unrelated process).

It is also significant that when we get the alert displayed about one tab
being blocked and we choose to close it as it does not seem to respond, it
will also kill all other tabs that are currently displaying a PNG image
(even if they are not active, and have no active script to update them, and
their images are already all loaded, the renderers are still active for the
case where we could switch to the other tab that would need to be
refreshed).

There may also be an incorrect pairing of GetDC/ReleaseDC after an
exception has occured between these two API calls. So this could be an
issue in one of your exception handlers that do not track and unwind
correctly the DC usage.

The sample code above (actually for Visual Basic) does not contain such
correct exception handler that will be needed in a multithreaded
environment : not only it does not check the value of the DC handled
returned and attempts to use it as is, but also any exception that could
occur between hDC=getDC(0) and ReleaseDC(0,hDC) is not protected by
a "finally" exception handler where ReleaseDC() should be called. Such
issue may exist in Chrome.

Codes related to this is in source file (changes that were performed for
integrating "Native CL" vs. windowed external renderers):

http://src.chromium.org/viewvc/chrome/trunk/src/chrome/renderer/render_process.cc?revision=27324&view=markup&pathrev=27324

Is there a way to disable the builtin (NaCL ?) renderer of PNG images and
use an external plugin instead ? This could help me to determine if this is
this integration of native windowless renderers (in worker threads rather
than separate processes) that is the source of this problem (i.e. internal
renderers use now an incorrect emulation of IPC mechanisms that were
initially built for external renderer processes).
--
Automated mail from issue updates at http://crbug.com/
Subscription options: http://groups.google.com/a/chromium.org/group/chromium-bugs
c***@googlecode.com
2010-12-03 16:16:13 UTC
Permalink
Comment #39 on issue 59315 by verdyp: Hangs, then crashes in Wikimedia
Commons (displaying an images gallery in the multilingual "Nature" category)
http://code.google.com/p/chromium/issues/detail?id=59315

Note also that the "LPK" module is loaded, and the "render_process.cc" code
checks for its presence like this:


RenderProcess::RenderProcess()
: ALLOW_THIS_IN_INITIALIZER_LIST(shared_mem_cache_cleaner_(
base::TimeDelta::FromSeconds(5),
this, &RenderProcess::ClearTransportDIBCache)),
sequence_number_(0) {
in_process_plugins_ = InProcessPlugins();
for (size_t i = 0; i < arraysize(shared_mem_cache_); ++i)
shared_mem_cache_[i] = NULL;

#if defined(OS_WIN)
// HACK: See http://b/issue?id=1024307 for rationale.
if (GetModuleHandle(L"LPK.DLL") == NULL) {
// Makes sure lpk.dll is loaded by gdi32 to make sure ExtTextOut() works
// when buffering into a EMF buffer for printing.
typedef BOOL (__stdcall *GdiInitializeLanguagePack)(int
LoadedShapingDLLs);
GdiInitializeLanguagePack gdi_init_lpk =
reinterpret_cast<GdiInitializeLanguagePack>(GetProcAddress(
GetModuleHandle(L"GDI32.DLL"),
"GdiInitializeLanguagePack"));
DCHECK(gdi_init_lpk);
if (gdi_init_lpk) {
gdi_init_lpk(0);
}
}
#endif
...

This is commented as a HACK, there should better be a trackable TODO
comment here ! It tries to explain this to solve Issue 1024307.
--
Automated mail from issue updates at http://crbug.com/
Subscription options: http://groups.google.com/a/chromium.org/group/chromium-bugs
c***@googlecode.com
2010-12-03 16:44:39 UTC
Permalink
Comment #40 on issue 59315 by verdyp: Hangs, then crashes in Wikimedia
Commons (displaying an images gallery in the multilingual "Nature" category)
http://code.google.com/p/chromium/issues/detail?id=59315

Another interesting comment in the last commit for this source also says:
"NaCl still cannot run in Chrome sandbox, the flag automatically disables
the sandboxCommitted". What this probably means is that when the NaCL
renderer hangs, it cannot be safely siolated to allow the rest of the
browser working properly. If it hangs and we get the alert for Chrome that
proposes to kill the load, multiple tabs will be killed (and this really
happens).

In fact, it can hangs the whole Windows desktop by grabbing its DC (and
probably not releasing it correctly after some exception).

Chrome is currently advertized as runinng plugins for separate tabs in
isolated processes, so that we can close them safely (but this is visibly
not the case for builtin image renderers, and it was also not the case for
the builtin Flash renderer and the builtin PDF renderer which have caused
similar hangs which seem to heve been solved now, probably by stronger
exception checking and recovery mechanisms).
--
Automated mail from issue updates at http://crbug.com/
Subscription options: http://groups.google.com/a/chromium.org/group/chromium-bugs
c***@googlecode.com
2010-12-03 17:00:48 UTC
Permalink
Comment #41 on issue 59315 by verdyp: Hangs, then crashes in Wikimedia
Commons (displaying an images gallery in the multilingual "Nature" category)
http://code.google.com/p/chromium/issues/detail?id=59315

One more question : why isn't the NaCL plugin listed in the "about:plugins"
with the list of other plugins and the MIME types or file extensions they
are handling ? I can have installed an alternate plugin for PNG images,
such as npqtplugin6.dll installed with QuickTime, but visibly, enabling or
disablling this plugin has no effect at all, and QuickTime is never used
for this MIME type.

We should be able to also disable the Chrome native plugin for specific
MIME types from the same page (notably: JPEG and PNG), as well as other
buitin renderers such as the spell checker, the Google software updater, or
the new audio/video renderer for HTML5 (apparently based on Ogg Theora), at
least to help seeing if the embedded renderers (that take the priority) are
the cause of the problem and also to find a workaround (including in case
of security issues).

Can we disable it at least via a command-line option when running
Chrome.exe ? And control how it works (single shared process or not,
enabling them to run in the network thread on Windows, instead of
serializing them to the main UI thread like in Linux and MacOSX) ?
--
Automated mail from issue updates at http://crbug.com/
Subscription options: http://groups.google.com/a/chromium.org/group/chromium-bugs
c***@googlecode.com
2011-03-16 17:16:33 UTC
Permalink
Comment #51 on issue 59315 by ***@gmail.com: Hangs, then crashes in
Wikimedia Commons (displaying an images gallery in the
multilingual "Nature" category)
http://code.google.com/p/chromium/issues/detail?id=59315


The problem still occurs (but more dramatically now in v10.0.648.133 or
today v10.0.648.134), since the installation of Windows 7 Service Pack 1
(which includes the new Win32 API about exception filters policy when
running Chrome as Win32 app over WOW64 in 64-bit OS).
The situation is worse because now when it hangs, it also hangs the whole
desktop, and all other open windows, that have apparently lots of
difficulties to refresh themselves (the refresh of ALL application windows
starts, very slowly, then hangs for a couple of minutes, then the whole
window becomes blank, and the refresh restarts again...)

Visibly this is caused by a very expensive use of GDI resources. And it is
very clearly related to the font renderer in Chrome, when it has to parse
long lists of fonts (from various complex scripts) in CSS: Chrome cannot
cope with pages that contain long lists of styles and various fonts for
distinct scripts or languages, and the text renderer is probably hang in a
loop, trying to use fallback fonts, and allocating GDI resources for
loading them indefinitely (and probably leaking them or allocating them in
long chains of recursive calls with many distainct duplicate GDI font
handles).

You might be interested to see that this is related to my personal CSS
preferences for Wikimedia that includes long lists of fonts that were
hevily tested and maintained for several years (including before I started
using Google Chrome 2+, in Firefox 4+, IE6+ and Opera 4+ without any
problem, not even any problem of performance).

This hang does not occur in Safari. It is specific to Chrome 4+ on all
versions of Windows, and must lie within some custom unsafe modifications
made in Webkit by Google for Chrome. In fact the same hang also occurs in
the opensourced Chromium version.

I have started booting Windows in kernel debug mode to see if I can get
more info about what is happening (I think that this is caused by some
unsafe exception management within callbacks from Windows GDI and/or
Uniscribe, or unsafe assumptions about how font fallbacks work in Windows).

I've tried to remove the two infinite loops you use
in "UniscribeHelper.cpp", using instead a single test when trying to
allocate and setup a DC (just like in examples shown in MSDN for
Uniscribe), and this compeltely solves the problem (I may see now
some "missing glyphs" for unsupported characters, or unexpected mixes of
font styles, but at least there's no more hang of the Windows GUI). Chrome
effectively is bogous there, and is contantly trying to allocating new
temporary DC when it hangs the whole desktop...

It seems that you have used an infinite loop only as an attempt to optimize
something that is in fact not so critical for performance (it just
optimizes code size with a few bytes, but definitely not execution time).
There's visibly an incorrect assumption that these loops will run only 1 or
2 times but never more. And this is now clearly wrong.

Now when running Chrome in Windows booted in kernel debug mode, I clearly
see many logged exceptions about excessive DC usage (possible leak detected
by Windows, that will then slow down Chrome just to giveback control to the
test of the OS and apps that need DC resources for their own display).

So please fix these infinite loops in Uniscribe.cpp, starting in lines 598+
and 791+. They are DEFINITELY WRONG. You absolutely don't need these loops,
even if the Uniscribe API returns E_PENDING (saying that you need to setup
and provide a temporary DC with a selected font, when Uniscribe autodetects
that the shaped text contains "complex" script).

You can (and should instead) assume that you'll always need to select the
font, and allocate and reuse the same DC with the same selected font for
shaping and rendering the text, and should assume that text will need to be
from complex scripts (requiring BiDi reordering and joining shaping, plus
font-dependant fallbacks, including through font linking).

But there's NO situation where you'll need to setup a temporary DC multiple
times for the same Uniscribe's ScriptShape() or ScripPlace() call with the
same text fragment (provided that you've correctly broken text fragments),
and in fact your code cannot even do that because it can only handle a
single predetermined "hfont" for that fragment.

But if you still want to use an infinite loop to simplify the code for
calling the Uniscribe API, please replace:

if (hr != E_PENDING)
break;

by:

if (hr != E_PENDING || tempDC != 0)
break;

or by:

if (hr != E_PENDING)
break;
if (tempDC != 0) {
DEBUG();
}

Note also that several successive runs of ScriptShape() or ScriptPlace()
will not necessarily use the same font (including within the same script,
due to the need for fallbacks), even if this occurs most of the time. For
this reason, you may cache the DC on which you can select any nomber of
fonts as often as needed between runs, without needing to save it and
restore the default font preselected in it (the whole Uniscribe shaping
will occur in the same thread and in the same DC).

However, you should already have determined the font to use in each run by
correctly splitting the text during word splitting (for justification
purpose) and script analysis. If you need to select a DC with that font,
you can reuse the same DC and noramlly don't need to reselect that font and
you'll get E_PENDING only once per text run.
--
Automated mail from issue updates at http://crbug.com/
Subscription options: http://groups.google.com/a/chromium.org/group/chromium-bugs
c***@googlecode.com
2011-09-08 10:52:51 UTC
Permalink
Comment #57 on issue 59315 by ***@gmail.com: Hangs, then crashes in
Wikimedia Commons (displaying an images gallery in the
multilingual "Nature" category)
http://code.google.com/p/chromium/issues/detail?id=59315

Not many fonts, just the fonts that are needed to support about 30 scripts.

And I doubt that this is GDI heap exhaustion (or if there's one, it's only
within the private heap of a single process, and it would not slow down all
other running apps and the desktop itself so dramatically, with lots of
difficulties to redraw the client area of their windows, including the icon
bar, even though there's absolutely NO problem in redraxing the window
decorations, or the large JPEG image on the desktop, or any icons on the
desktop, which appear with their normal transparencies, crisp fonts still
rendered with ClearType antialiasing, and no loss of colors or brush styles
and 3D light effects).

Note also that I have plenty of RAM, and plenty of disk space. I never
notice any sudden activity goind to swap plenty of memory pages to/from the
swapfile on disk.

As well if a video is running in the background, it continues and is not
affected, the same is true for a music playng in the background, including
when listening an MP3 webradio.

What is really affected is the attempt to change the desktop geometries,
i.E. the layout of windows on top of them, as if there was some window
constantly trying to bring up to the top of stack and immediately hiding
without anything visible: all other windows are being refreshed constantly,
and if there's a heap overhead, it's not in the GDI heap but most probably
in the USER32 heap (that manage the stack of windows, for custom buttons in
dialog forms, popup menus or alerts, and tooltips).

And I repeat it: only ONE thread in ONE process is running at 100% of a
SINGLE thread of a SINGLE core of the CPU. All other cores in the CPU are
alsmost at 0% time used: one thread apparently blocks all the other threads
of running, meaning that it is courrently holding an exclusive lock on a
critical shared resource of the desktop. My opinion is that this critical
resource if the handle of the window for the desktop itself, which is never
released or when it is, there's a running priority that occurs so that the
salvage thread takes takes immediately the control again due to its high
priority.

And yes, I suspect that this occurs in the "strange" infinite loop that is
used in the Chrome's code for the OpenType/Uniscribe shaping and rendering
engine for drawing international texts.

Your suggestion with "--disable-winsta" has absolutely no effect, the same
bug occurs immediately (on all the PC's where I have tested it with Chrome,
using various versions of Windows).

Anyway I'm still not sure that this is caused by the OpenType renderer, as
I still hesitate with the way Chrome forks a new rendering task with its
background processes (it could be a bug in the complex IPC mechanism it
uses to create its isolation "sandboxes": memory map handles ? posted
windows events ?).
--
Automated mail from issue updates at http://crbug.com/
Subscription options: http://groups.google.com/a/chromium.org/group/chromium-bugs
c***@googlecode.com
2011-09-12 04:55:31 UTC
Permalink
Updates:
Status: Assigned
Owner: ***@chromium.org

Comment #58 on issue 59315 by ***@chromium.org: Hangs, then crashes in
Wikimedia Commons (displaying an images gallery in the
multilingual "Nature" category)
http://code.google.com/p/chromium/issues/detail?id=59315

Issue 96021 is most likely a related (duplicate?) problem...

As described in there, the tight loop getting the screen DC and calling
SelectObject with a Font on it would explain the every app refresh
problem...

But I wasn't able to reproduce the problem with the URL mentioned in the
initial comment... I'm guessing something else on the machine is getting in
the way... :-/... But Chrome is definitely at fault in the tight loop for
font shaping... Though it would be nice to be able to reproduce it without
having to exhaust the GDI handle count first... There might be something
else that get the ScriptShape/ScriptePlace calls to always return
E_PENDING...

Any ideas would be welcome...
--
Automated mail from issue updates at http://crbug.com/
Subscription options: http://groups.google.com/a/chromium.org/group/chromium-bugs
c***@googlecode.com
2011-09-12 05:43:54 UTC
Permalink
Comment #59 on issue 59315 by ***@gmail.com: Hangs, then crashes in
Wikimedia Commons (displaying an images gallery in the
multilingual "Nature" category)
http://code.google.com/p/chromium/issues/detail?id=59315

It may help if you just used my custom CSS file that has been set in my own
Wikimedia profiles. Note the long (but carefully sorted) list of fallback
fonts.
The bug does not occur always, but more frequently when I am logged on
Wikimedia.
See:

http://fr.wikipedia.org/wiki/Utilisateur:Verdy_p/polices.css

This CSS file is shared and "@import"ed from other Wikimedia sites, from my
local monobook.css or vector.css files, like this one for my account on
Commons:

http://commons.wikimedia.org/wiki/User:Verdy_p/monobook.css

You may wonder why I have so long lists, but I'm involved in wroking with
lots of translations (on Commons or in Wiktionary) written in various
scripts (including on the same page).

Most often there's no problem, but the bug still occurs always in
the "Category:Nature" page of Commons.

Each time, Chrome hangs immediately with an all-white window and only the
status bar showng "Waiting for <server>.wikimedia.org", the server name
varying between the local wiki for javascripts, or the image server. (the
status bar is correctly refreshed without problem, only the renderer does
not complete correctly, and uses 100% of a CPU thread in one core, staling
almost all other CPU cores or threads, so that only the Chrome renderer
thread gets significant processing time).

Chrome will hang much more easily if all the HTML pages with its referenced
scripts, stylesheets and images are already preloaded in the cache (for
example when killing Chrome, and restarting it). It occurs much more often
in Commons only because there are lots of thumbnail images in a category
page, and there are also frequent translations in many scripts.

Thanks for *finally* admitting that the infinite loop that expects that
E_PENDING occurs only once is certainly a problem, or at least not very
safe (and in addition, even if it stops, there's still the own Chrome's
font fallback mechanism, and the interaction with the CSS font families:
there may be a bug when processing these lists of fonts due to mutual font
fallbacks exhisting in Windows with Uniscribe).

I can reproduce the problem on ALL Windows machines I have been able to use
(not just mine), with ALL installed versions of Chrome since betas of 4.0,
and with ALL versions of Windows, and ALL types of graphics cards (ATI,
nVidia, Intel, ...), even very old ones that are no longer sold, or in
virtual environment with a virtual only emulation for remote sessions,
using Microsoft only virtual display drivers. It also occurs when using the
Microsoft SVGA default driver (without any hardware acceleration), for
example when booting Windows in safe mode with network support (yes it is
somewhat slower to render Internet pages, but it is perfectly usable with
modern processors for simple browsing of pages without many animations or
videos or shading effects; but Chrome will also hang exactly the same way).

It occurs also independantly of if the PC is connected via a VPN, or an
hardware Ethernet PCI board, or a Wifi PCI board, or a Wifi USB adapter.
--
Automated mail from issue updates at http://crbug.com/
Subscription options: http://groups.google.com/a/chromium.org/group/chromium-bugs
c***@googlecode.com
2011-09-12 06:00:01 UTC
Permalink
Comment #60 on issue 59315 by ***@gmail.com: Hangs, then crashes in
Wikimedia Commons (displaying an images gallery in the
multilingual "Nature" category)
http://code.google.com/p/chromium/issues/detail?id=59315

Note that my comments 37 and 38 above identified the GetDC(0) problem since
long. As well as the tight loop, potentially infinite.

If you call GetDC(0) multiple times, you will not get multiple handles
though, the same should be returned to the same process, but a counter will
be incremented internally by Windows, so that you have to ReleaseDC() the
same number of times (MSDN clearly says that GetDC() and ReleaseDC() must
absolutely be paired).

But can't a test be performed to know if we already have a non-NULL DC
handle before calling GetDC(0) within the loop ? That's what I suggested
since long !
--
Automated mail from issue updates at http://crbug.com/
Subscription options: http://groups.google.com/a/chromium.org/group/chromium-bugs
c***@googlecode.com
2011-09-13 00:34:40 UTC
Permalink
Comment #81 on issue 59315 by ***@gmail.com: Hangs, then crashes in
Wikimedia Commons (displaying an images gallery in the
multilingual "Nature" category)
http://code.google.com/p/chromium/issues/detail?id=59315

Note; there are THREE tight loops:
- (1) one in UniscribeHelper::fillRuns(), to itemize the string into runs
of glyhps with the same minimal set of font attributes or resolved script
direction,
- (2) one in UniscribeHelper::shape() to perform locate cmapped glyphs in
fonts and determining font fallbacks, and then compute substitutions and
ligatures and shaping,
- (3) and one in UniscribeHelper::fillShapes() that computes the
positioning, kerning, and determine justifications (and length of kashidas)
once we have computed the advance widths of glyphs.

The final drawing of positioned glyphs is made elsewhere, after the glyphs
have been reordered according to the Bidi algorithm (the code assumes at
this point that the itemized runs are unidirectional, as determined during
the first step).

Eah of these 3 loops may return E_PENDING as an indicator of the fact that
the result is font dependant.

The code assumes that E_PENDING will not be returned indefinitely in the
2nd and 3rd step; but the error condition that occurs in step 2
(UniscribeHelper::shape) when no more fonts are available (when determining
font fallbacks) is not properly checked, and if it occurs, the step 1
(itemization) should be performed once again, or the run should be
shortened, otherwise step 3 will still proceed and will generate the
infinite loop.

(it seems to me that if step 2, i.e. shaping, fails with the last tried
font, the simple path should be used instead to render the itemized run
without Uniscribe shaping and positioning).
--
Automated mail from issue updates at http://crbug.com/
Subscription options: http://groups.google.com/a/chromium.org/group/chromium-bugs
c***@googlecode.com
2011-10-23 12:25:37 UTC
Permalink
Comment #92 on issue 59315 by ***@gmail.com: Hangs, then crashes in
Wikimedia Commons (displaying an images gallery in the
multilingual "Nature" category)
http://code.google.com/p/chromium/issues/detail?id=59315

See createFontIndirectAndGetWinName(), and other code related to the font
cache...
--
Automated mail from issue updates at http://crbug.com/
Subscription options: http://groups.google.com/a/chromium.org/group/chromium-bugs
c***@googlecode.com
2011-10-23 17:40:36 UTC
Permalink
Comment #93 on issue 59315 by ***@gmail.com: Hangs, then crashes in
Wikimedia Commons (displaying an images gallery in the
multilingual "Nature" category)
http://code.google.com/p/chromium/issues/detail?id=59315

There's another bug in this new implementation, you want to substitute the
API's ScriptItemizeOpenType() and ScriptShapeOpenType() instead of
ScriptItemize() and ScriptShape(), depending on the verson of USP10.DLL,
but you don't do that for ScriptPlace().

The MSDN doc clearly says that the specialized API for OpenType and the
generic API must *NOT* be mixed. The runs determined in fillRuns(), and
then compressed by removing some "unknown script" runs, as determined by
the OpenType specialized API is definitely not compatible with the generic
ScriptPlace() call.

So if you use the dynamic loader for the gScriptItemizeOpenTypeFunc and
gScriptShapeOpenTypeFunc, you must *also* use a dynamic loader for the
gScriptPlaceOpenTypeFunc ! And the "optimization" to "pack consecutive
runs" made after calling gScriptItemizeOpenTypeFunc() in lines 574-588 is
most probably not safe as well. Did you ever test it separately in a basic
test application ?
--
Automated mail from issue updates at http://crbug.com/
Subscription options: http://groups.google.com/a/chromium.org/group/chromium-bugs
c***@googlecode.com
2011-10-23 17:48:40 UTC
Permalink
Comment #94 on issue 59315 by ***@gmail.com: Hangs, then crashes in
Wikimedia Commons (displaying an images gallery in the
multilingual "Nature" category)
http://code.google.com/p/chromium/issues/detail?id=59315

Note also that the specialized ScriptXXXOpenType() functions of the Windows
API are not portable. OpenType is not the only font technology used in
Windows (Windows also supports other font managers, and there may even be
some legacy fonts that are definitely not compatible with OpenType, such
as .fon bitmap fonts, inclusing system fonts that may be used in some
fallbacks). The Adobe Type Manager installs its own font manager
compatible with Uniscribe but not with the specific ScriptXXXXOpenType()
calls. Windows will also likely add other font technologies (for example
SVG fonts); you may also have support on the system for other technologies
such as SIL Graphite.

Do you really need ScriptXXXOpenType() ???
--
Automated mail from issue updates at http://crbug.com/
Subscription options: http://groups.google.com/a/chromium.org/group/chromium-bugs
c***@googlecode.com
2011-10-23 17:56:44 UTC
Permalink
Comment #95 on issue 59315 by ***@gmail.com: Hangs, then crashes in
Wikimedia Commons (displaying an images gallery in the
multilingual "Nature" category)
http://code.google.com/p/chromium/issues/detail?id=59315

Note for example that ScriptXXXXOpenType() APIs of Win32 are not supported
in Windows Embedded Compact environments, based on .Net which generalizes
the API. these APIs should be considered internal for implementing the
generic (non-OpenType specific ) portable Uniscribe APIs.

See the note at end of:
http://msdn.microsoft.com/en-us/library/windows/desktop/dd317792(v=VS.85).aspx

[quote]
As shown in the following table, Uniscribe version 1.6 or greater supports
several functions that take advantage of OpenType tags. They can be
substituted for the corresponding regular Uniscribe functions. Generally
your applications should work entirely with functions from one set or the
other and should not attempt to "mix and match" functions.

Regular Uniscribe function Equivalent OpenType function
ScriptItemize ScriptItemizeOpenType
ScriptShape ScriptShapeOpenType
ScriptPlace ScriptPlaceOpenType
[/quote]
--
Automated mail from issue updates at http://crbug.com/
Subscription options: http://groups.google.com/a/chromium.org/group/chromium-bugs
c***@googlecode.com
2011-10-23 18:09:56 UTC
Permalink
Comment #96 on issue 59315 by ***@gmail.com: Hangs, then crashes in
Wikimedia Commons (displaying an images gallery in the
multilingual "Nature" category)
http://code.google.com/p/chromium/issues/detail?id=59315

So now I'm convinced that thuis is the call to SriptPlace instead of a
dynamically loaded ScriptPlaceOpenTypeFunc which is causing the massive
resource leakage, where the ScriptPlace() call fails to place runs that
have been itemized with the extended API and unsafely compressed in lines
574-588...

Note that in the .Net and Compact mobile environments, there's no longer
two APIs, only one already compatible with OpenType support. Consider
the "Regular Uniscribe function" set as a legacy for older versions of
Windows for desktops. It may happen that the bug we get now only occurs in
desktop versions of Windows using Uniscribe 1.6 or later.
--
Automated mail from issue updates at http://crbug.com/
Subscription options: http://groups.google.com/a/chromium.org/group/chromium-bugs
c***@googlecode.com
2011-10-23 18:20:02 UTC
Permalink
Comment #97 on issue 59315 by ***@gmail.com: Hangs, then crashes in
Wikimedia Commons (displaying an images gallery in the
multilingual "Nature" category)
http://code.google.com/p/chromium/issues/detail?id=59315

So effectively, the infinite lop occurs in lines 846-860 because you have
used the "mix and match" trick, and unsafely attempted to make it work with
the compression trick in lines 574-588 of the UniscribeHelper.cpp source;
the ScriptPlace() call fails because there are runs with unknown scripts
(such as generic punctuations or symbols) that have still been itemized
separately by ScriptItemizeOpenType() because they needed a separate font
or separate fallback. They were itemized because the OpneType version of
Uniscribe determined that there was the need for a complex path for
*placing* glyphs, even if there was no apparent complex path for the glyph
*shaping* pass.

ScriptPlace() then fails repeatedly: the compressed run can no longer be
placed correctly as they need to use separate fonts... And this causes
havoc in pages mixing complex scripts like Indic/Tibetan/Khmer/Lao, and/or
RTL scripts requiring specific placement and mirroring rules like
Arabic/Hebrew, and/or "simple" LTR generic scripts like
Latin/Greek/Cyrillic/Armenian/Georgian and Han.
--
Automated mail from issue updates at http://crbug.com/
Subscription options: http://groups.google.com/a/chromium.org/group/chromium-bugs
c***@googlecode.com
2011-10-23 18:35:09 UTC
Permalink
Comment #98 on issue 59315 by ***@gmail.com: Hangs, then crashes in
Wikimedia Commons (displaying an images gallery in the
multilingual "Nature" category)
http://code.google.com/p/chromium/issues/detail?id=59315

The compression used in lines 574-588 is clearly incorrect.
See the description of SCRIPT_TAG_UNKNOWN (defined as ((ULONG)0x00000000))
http://msdn.microsoft.com/en-us/library/dd368557(VS.85).aspx

[quote]
pScriptTags [out]
Pointer to a buffer in which the function retrieves an array of
OPENTYPE_TAG structures representing script tags. The buffer should be
cMaxItems * sizeof(OPENTYPE_TAG) bytes in length.

Note When all characters in an item are neutral, the value of this
parameter is SCRIPT_TAG_UNKNOWN (0x00000000). This can happen, for example,
if an item consists entirely of punctuation.
[/quote]

so effectively you will compress runs of generic punctuations into the same
run as a scrip-specific run or which another font (or fallback font) has
been determined, which may not be the one determined for the punctuations
run.

Runs of generic punctuations (SCRIPT_TAG_UNKNOWN) generally need not any
shaping, i.e. OpenType "GSUB" features (but this is *not* true with RTL
scripts due to mirroring behaviors, plus the effects of the Unicode Bidi
algorithm), but may still need glyph placing, i.e. OpenType "GPOS" features
and kerning.
--
Automated mail from issue updates at http://crbug.com/
Subscription options: http://groups.google.com/a/chromium.org/group/chromium-bugs
c***@googlecode.com
2011-10-23 18:53:16 UTC
Permalink
Comment #99 on issue 59315 by ***@gmail.com: Hangs, then crashes in
Wikimedia Commons (displaying an images gallery in the
multilingual "Nature" category)
http://code.google.com/p/chromium/issues/detail?id=59315

In other words, you MUST absolutely fix this function:

68 static void loadOpenTypeFunctions()
69 {
70 HMODULE hModule = GetModuleHandle(L"usp10");
71 if (hModule) {
72 gScriptItemizeOpenTypeFunc =
reinterpret_cast<ScriptItemizeOpenTypeFunc>(GetProcAddress(hModule, "ScriptItemizeOpenType"));
73 gScriptShapeOpenTypeFunc =
reinterpret_cast<ScriptShapeOpenTypeFunc>(GetProcAddress(hModule, "ScriptShapeOpenType"));
74 }
75 if (!gScriptItemizeOpenTypeFunc || !gScriptShapeOpenTypeFunc) {
76 gScriptItemizeOpenTypeFunc = 0;
77 gScriptShapeOpenTypeFunc = 0;
78 }
79 gOpenTypeFunctionsLoaded = true;
80 }

by defining and initializing gScriptPlaceOpenTypeFunc and testing it: all
three functions must be present (and tested and used in line 848, like in
line 567 for gScriptItemizeOpeenTypeFunc, and like in line 670 for
gScriptShapeOpenTypeFunc), or none of them must be used and all must be set
to 0 if any one is missing.

With this addition, you'll be later have the possibility to really use the
more advanced OpenType features (that are almost all disabled in the
initialized SCRIPT_CONTROL and SCRIPT_STATE structures in
UniscribeHelper::fillRuns() in lines 514-543, notably ligatures, and
pre-bound and post-bound shaping for the breaking behavior of cursive
scripts).

If you really want to merge neutral items (SCRIPT_TAG_UNKNOWN), instead of
using your compression trick used after calling gSScriptItemizeOpenType(),
set the SCRIPT_CONTROL::fMergeNeutralItems field to 1 in the
initialization, and it will be much safer (but you'll loose some support to
fallback fonts, and some punctuations occuring just after a run in a
complex script like Khmer or Thai may now appear as empty boxes depending
on the coverage of fonts installed on the system)...
--
Automated mail from issue updates at http://crbug.com/
Subscription options: http://groups.google.com/a/chromium.org/group/chromium-bugs
c***@googlecode.com
2011-11-28 21:27:07 UTC
Permalink
Updates:
Labels: -WebKit-ID-68598-ASSIGNED WebKit-ID-68598-RESOLVED
WebKit-Rev-101059

Comment #105 on issue 59315 by ***@chromium.org: Hangs, then crashes
in Wikimedia Commons (displaying an images gallery in the
multilingual "Nature" category)
http://code.google.com/p/chromium/issues/detail?id=59315#c105

https://bugs.webkit.org/show_bug.cgi?id=68598
http://trac.webkit.org/changeset/101059
--
Automated mail from issue updates at http://crbug.com/
Subscription options: http://groups.google.com/a/chromium.org/group/chromium-bugs
Loading...