[coreboot] Wired problems with Intel skylake based board

[Christian Gmeiner]

Hi all

I have an almost working coreboot/u-boot boot solution based on
kabylake FSP. Most of the time
the system works as expected but from time to rebooting the system
fails completely. On my board,
which is powered by an COM Express module (i3-6100U), there are two
FPGAs connected via PCIe to the
system where one is used to reset the system. The reset is done via SYS_RESET#.

Now I run into different kind of issues:
- pcie link training fails from time to time multiple times with the
end result at FSP does multiple
global resets.
- looks like PLTRST# of the sunrisepoint pch holds the system in reset
for minutes.

Are there any hints to debug my issues?
--
greets
--
Christian Gmeiner, MSc

https://christian-gmeiner.info

[Peter Stuge]

Most of the time the system works as expected but from time to rebooting
the system fails completely.

Only ever when rebooting, or does cold boot also fail sometimes?

(Make a test system to cold boot your system in a loop.)

there are two FPGAs connected via PCIe to the system where one is used
to reset the system. The reset is done via SYS_RESET#.

Are the FPGAs also reset by that?

If yes, how long do they need to initialize to where HDL acts
correctly on PCIe?

If no, how long do they need to move from resetting the system to
where HDL acts correctly on PCIe for the newly resetted platform?

- pcie link training fails from time to time

On both links, or only one of them? Can you tell?

- looks like PLTRST# of the sunrisepoint pch holds the system in reset
for minutes.

I don't know if there's enough PCH documentation to know exactly why
it would do that - but I can imagine that it holds reset as long as
some conditions are not met, I can also imagine that the FPGAs cause
some undefined PCIe behavior in the PCH which happens to get it stuck
in reset for a while.

Are there any hints to debug my issues?

As always, try to isolate the problem.

Can you completely remove one or ideally both FPGAs from the equation?

You mention that one of them is used for reset. At least disable the
other one, destructively if need be.


//Peter

[Christian Gmeiner]

Post by Peter Stuge

Most of the time the system works as expected but from time to rebooting
the system fails completely.

Only ever when rebooting, or does cold boot also fail sometimes?

Both fail.

Post by Peter Stuge
(Make a test system to cold boot your system in a loop.)

there are two FPGAs connected via PCIe to the system where one is used
to reset the system. The reset is done via SYS_RESET#.

Are the FPGAs also reset by that?

Yes

Post by Peter Stuge
If yes, how long do they need to initialize to where HDL acts
correctly on PCIe?

I would need to measure this but i would say less then 300ms.

Post by Peter Stuge

- pcie link training fails from time to time

On both links, or only one of them? Can you tell?

This is hard to tell.. all I see that the link gets reset quite fast
and quite often.

Post by Peter Stuge

- looks like PLTRST# of the sunrisepoint pch holds the system in reset
for minutes.

I don't know if there's enough PCH documentation to know exactly why
it would do that - but I can imagine that it holds reset as long as
some conditions are not met, I can also imagine that the FPGAs cause
some undefined PCIe behavior in the PCH which happens to get it stuck
in reset for a while.

Are there any hints to debug my issues?

As always, try to isolate the problem.
Can you completely remove one or ideally both FPGAs from the equation?
You mention that one of them is used for reset. At least disable the
other one, destructively if need be.

During the last weeks I found the root cause of my problem - PCIe
spread spectrum

Our FPGAs need a stable 100MHz PCIE clock to work. The used FSP config
thing looked
like this:

void mainboard_memory_init_params(FSPM_UPD *mupd)
{
FSP_M_CONFIG *mem_cfg;
struct spd_block blk = {
.addr_map = { 0x50 },
};

mem_cfg = &mupd->FspmConfig;

mem_cfg->PegDisableSpreadSpectrumClocking = 1;
mem_cfg->PchPmPciePllSsc = 0;

...
}

With this configuration the PCIe reference clock was off more then 8% which
caused the system to hang during cold and warm boots.

In the next step I removed assignment of PchPmPciePllSsc as it is documented
as 'No BIOS override'. With this change I got more then 1000 soft and
2000 hard reboots
without any problem. Keep in mind we started with only 10 successful reboots.

The big problem is that PegDisableSpreadSpectrumClocking has no effect
at all. I measured
the freq it is not the 100MHz as expected. And I need to have a stable
100MHz this clock source
is used internally by the FPGA to drive internal clocks. The end
results is that EtherCAT is not
able to sync.

I tried to change the ICC config via MEI messaging but I am not able
to change the clock settings
even with an successful return code.
--
greets
--
Christian Gmeiner, MSc

https://christian-gmeiner.info

[Nico Huber]

Post by Christian Gmeiner
During the last weeks I found the root cause of my problem - PCIe
spread spectrum
Our FPGAs need a stable 100MHz PCIE clock to work. The used FSP config
thing looked
void mainboard_memory_init_params(FSPM_UPD *mupd)
{
FSP_M_CONFIG *mem_cfg;
struct spd_block blk = {
.addr_map = { 0x50 },
};
mem_cfg = &mupd->FspmConfig;
mem_cfg->PegDisableSpreadSpectrumClocking = 1;
mem_cfg->PchPmPciePllSsc = 0;
...
}
With this configuration the PCIe reference clock was off more then 8% which
caused the system to hang during cold and warm boots.
In the next step I removed assignment of PchPmPciePllSsc as it is documented
as 'No BIOS override'. With this change I got more then 1000 soft and
2000 hard reboots
without any problem. Keep in mind we started with only 10 successful reboots.

Please be more specific about the final setting of this UPD. `No BIOS
override` is the documentation for the default value of 0xff. But is
this set to the default in the binary? who knows...

Post by Christian Gmeiner
The big problem is that PegDisableSpreadSpectrumClocking has no effect
at all. I measured
the freq it is not the 100MHz as expected. And I need to have a stable
100MHz this clock source
is used internally by the FPGA to drive internal clocks. The end
results is that EtherCAT is not
able to sync.

This setting is about a different clock, I guess. Can you please clarify
what is connected to which clock on your board.

Nico

[Christian Gmeiner]

Post by Nico Huber

Post by Christian Gmeiner
During the last weeks I found the root cause of my problem - PCIe
spread spectrum
Our FPGAs need a stable 100MHz PCIE clock to work. The used FSP config
thing looked
void mainboard_memory_init_params(FSPM_UPD *mupd)
{
FSP_M_CONFIG *mem_cfg;
struct spd_block blk = {
.addr_map = { 0x50 },
};
mem_cfg = &mupd->FspmConfig;
mem_cfg->PegDisableSpreadSpectrumClocking = 1;
mem_cfg->PchPmPciePllSsc = 0;
...
}
With this configuration the PCIe reference clock was off more then 8% which
caused the system to hang during cold and warm boots.
In the next step I removed assignment of PchPmPciePllSsc as it is documented
as 'No BIOS override'. With this change I got more then 1000 soft and
2000 hard reboots
without any problem. Keep in mind we started with only 10 successful reboots.

Please be more specific about the final setting of this UPD. `No BIOS
override` is the documentation for the default value of 0xff. But is
this set to the default in the binary? who knows...

void mainboard_memory_init_params(FSPM_UPD *mupd)
{
FSP_M_CONFIG *mem_cfg;
struct spd_block blk = {
.addr_map = { 0x50 },
};

mem_cfg = &mupd->FspmConfig;

/* Disable PCIe Spread Spectrum Clocking */
printk(BIOS_ERR, "PegDisableSpreadSpectrumClocking: %x\n",
mem_cfg->PegDisableSpreadSpectrumClocking);
printk(BIOS_ERR, "PchPmPciePllSsc: %x\n", mem_cfg->PchPmPciePllSsc);
mem_cfg->PegDisableSpreadSpectrumClocking = 1;

get_spd_smbus(&blk);
dump_spd_info(&blk);
assert(blk.spd_array[0][0] != 0);

mainboard_fill_dq_map_data(&mem_cfg->DqByteMapCh0);
mainboard_fill_dqs_map_data(&mem_cfg->DqsMapCpu2DramCh0);
mainboard_fill_rcomp_res_data(&mem_cfg->RcompResistor);
mainboard_fill_rcomp_strength_data(&mem_cfg->RcompTarget);

mem_cfg->DqPinsInterleaved = TRUE;
mem_cfg->MemorySpdDataLen = blk.len;
mem_cfg->MemorySpdPtr00 = (uintptr_t) blk.spd_array[0];
}

And here is the output taken from cbmem -1:

..
FMAP: base = ff000000 size = 1000000 #areas = 4
FMAP: area RW_MRC_CACHE found @ a50000 (65536 bytes)
MRC: no data in 'RW_MRC_CACHE'
PegDisableSpreadSpectrumClocking: 0
PchPmPciePllSsc: ff
SPD @ 0x50
SPD: module type is DDR4
..

Post by Nico Huber

Post by Christian Gmeiner
The big problem is that PegDisableSpreadSpectrumClocking has no effect
at all. I measured
the freq it is not the 100MHz as expected. And I need to have a stable
100MHz this clock source
is used internally by the FPGA to drive internal clocks. The end
results is that EtherCAT is not
able to sync.

This setting is about a different clock, I guess. Can you please clarify
what is connected to which clock on your board.

Our FPGAs are using the 100 MHz PCIe clock as input to drive internal
clocks etc. One
of these clock is used for EtherCAT. If spread spectrum is active we
are around 100 MHz
(worst measured freq was ~92 MHz) and as a result the internal FPGA clock is not
stable/reliable --> EtherCAT sync fails.

If I use the following pattern:
mem_cfg->PegDisableSpreadSpectrumClocking = 1;
mem_cfg->PchPmPciePllSsc = 0;

I get a stable 100 MHz PCIe clock signal and everything works - except
the device hangs
after < 10 warm reboots. Looks like PCIe link training fails
uncountable times (seen with protocol
analyzer).
--
greets
--
Christian Gmeiner, MSc

https://christian-gmeiner.info

[Naresh G. Solanki]

Hi,

There are two things here.
1. System fails to boot i.e., hangs
2. FPGA's connected to root port are not detected in FW/OS

For problem 1, can you give data on where exactly it hangs?, Is it in
OS or FW ?, Can you provide kernel/coreboot log, port 80 dump when it
hangs.
For problem 2, Can you try setting UPD PcieRpHotPlug, & check the
behaviour. reference:
https://review.coreboot.org/cgit/coreboot.git/tree/src/soc/intel/skylake/chip_fsp20.c#n300

Can you please check FPGA datasheet & check for frequency tolerance of
PCIE_CLK_REF signal from FPGA.
Also can you re-verify board layout for any PDG(Platform Design Guide)
violations like impedance, length matching, limits for differential
PCIE_CLK_REF, PCIe Lanes.
Sometimes noise generated within board can generate huge EMI. I assume
high noisy circuits(power supply, backlight driver etc) are kept away
from high speed PCIe signals.

Regards,
Naresh G Solanki
On Mon, Oct 15, 2018 at 5:37 PM Christian Gmeiner

Post by Christian Gmeiner

Post by Nico Huber

Post by Christian Gmeiner
During the last weeks I found the root cause of my problem - PCIe
spread spectrum
Our FPGAs need a stable 100MHz PCIE clock to work. The used FSP config
thing looked
void mainboard_memory_init_params(FSPM_UPD *mupd)
{
FSP_M_CONFIG *mem_cfg;
struct spd_block blk = {
.addr_map = { 0x50 },
};
mem_cfg = &mupd->FspmConfig;
mem_cfg->PegDisableSpreadSpectrumClocking = 1;
mem_cfg->PchPmPciePllSsc = 0;
...
}
With this configuration the PCIe reference clock was off more then 8% which
caused the system to hang during cold and warm boots.
In the next step I removed assignment of PchPmPciePllSsc as it is documented
as 'No BIOS override'. With this change I got more then 1000 soft and
2000 hard reboots
without any problem. Keep in mind we started with only 10 successful reboots.

Please be more specific about the final setting of this UPD. `No BIOS
override` is the documentation for the default value of 0xff. But is
this set to the default in the binary? who knows...

void mainboard_memory_init_params(FSPM_UPD *mupd)
{
FSP_M_CONFIG *mem_cfg;
struct spd_block blk = {
.addr_map = { 0x50 },
};
mem_cfg = &mupd->FspmConfig;
/* Disable PCIe Spread Spectrum Clocking */
printk(BIOS_ERR, "PegDisableSpreadSpectrumClocking: %x\n",
mem_cfg->PegDisableSpreadSpectrumClocking);
printk(BIOS_ERR, "PchPmPciePllSsc: %x\n", mem_cfg->PchPmPciePllSsc);
mem_cfg->PegDisableSpreadSpectrumClocking = 1;
get_spd_smbus(&blk);
dump_spd_info(&blk);
assert(blk.spd_array[0][0] != 0);
mainboard_fill_dq_map_data(&mem_cfg->DqByteMapCh0);
mainboard_fill_dqs_map_data(&mem_cfg->DqsMapCpu2DramCh0);
mainboard_fill_rcomp_res_data(&mem_cfg->RcompResistor);
mainboard_fill_rcomp_strength_data(&mem_cfg->RcompTarget);
mem_cfg->DqPinsInterleaved = TRUE;
mem_cfg->MemorySpdDataLen = blk.len;
mem_cfg->MemorySpdPtr00 = (uintptr_t) blk.spd_array[0];
}
..
FMAP: base = ff000000 size = 1000000 #areas = 4
MRC: no data in 'RW_MRC_CACHE'
PegDisableSpreadSpectrumClocking: 0
PchPmPciePllSsc: ff
SPD: module type is DDR4
..

Post by Nico Huber

Post by Christian Gmeiner
The big problem is that PegDisableSpreadSpectrumClocking has no effect
at all. I measured
the freq it is not the 100MHz as expected. And I need to have a stable
100MHz this clock source
is used internally by the FPGA to drive internal clocks. The end
results is that EtherCAT is not
able to sync.

This setting is about a different clock, I guess. Can you please clarify
what is connected to which clock on your board.

Our FPGAs are using the 100 MHz PCIe clock as input to drive internal
clocks etc. One
of these clock is used for EtherCAT. If spread spectrum is active we
are around 100 MHz
(worst measured freq was ~92 MHz) and as a result the internal FPGA clock is not
stable/reliable --> EtherCAT sync fails.
mem_cfg->PegDisableSpreadSpectrumClocking = 1;
mem_cfg->PchPmPciePllSsc = 0;
I get a stable 100 MHz PCIe clock signal and everything works - except
the device hangs
after < 10 warm reboots. Looks like PCIe link training fails
uncountable times (seen with protocol
analyzer).
--
greets
--
Christian Gmeiner, MSc
https://christian-gmeiner.info
--
https://mail.coreboot.org/mailman/listinfo/coreboot

[Christian Gmeiner]

Hi

Am Mo., 15. Okt. 2018 um 15:26 Uhr schrieb Naresh G. Solanki

Post by Naresh G. Solanki
Hi,
There are two things here.
1. System fails to boot i.e., hangs

Correct.

Post by Naresh G. Solanki
2. FPGA's connected to root port are not detected in FW/OS

Wrong. The FPGAs are detected correctly and work most of the time. The only
problem we have is that the PCIe reference clock is not 100MHz as
spread spectrum
is activated. This causes that the internal clock used for EtherCAT
ist not reliable
which has the result that we fail to sync with EtherCAT clients and
there no communication
over EtherCAT works.

Post by Naresh G. Solanki
For problem 1, can you give data on where exactly it hangs?, Is it in
OS or FW ?, Can you provide kernel/coreboot log, port 80 dump when it
hangs.

It only hangs If I change the following values:

mem_cfg->PegDisableSpreadSpectrumClocking = 1;
mem_cfg->PchPmPciePllSsc = 0;

The 'physical' cause for the hang is also known: PLT_RST# never gets
high again. There are chances
that it does not hang but PCI devices (sata, usb hc, ...) are not
working as expected as the PCe reference clock
is in such a case at around 92 Mhz.

The end goal is to disable PCIe Spread Specturm and get a constant
PCIe reference clock
of 100MHz.

The system does not hang if I only change

mem_cfg->PegDisableSpreadSpectrumClocking = 1;

But it has no effect on the PCIe reference clock and it looks like
spread spectrum is still used.

Post by Naresh G. Solanki
For problem 2, Can you try setting UPD PcieRpHotPlug, & check the
https://review.coreboot.org/cgit/coreboot.git/tree/src/soc/intel/skylake/chip_fsp20.c#n300
Can you please check FPGA datasheet & check for frequency tolerance of
PCIE_CLK_REF signal from FPGA.
Also can you re-verify board layout for any PDG(Platform Design Guide)
violations like impedance, length matching, limits for differential
PCIE_CLK_REF, PCIe Lanes.
Sometimes noise generated within board can generate huge EMI. I assume
high noisy circuits(power supply, backlight driver etc) are kept away
from high speed PCIe signals.
Regards,
Naresh G Solanki
On Mon, Oct 15, 2018 at 5:37 PM Christian Gmeiner

Post by Christian Gmeiner

Post by Nico Huber

Post by Christian Gmeiner
During the last weeks I found the root cause of my problem - PCIe
spread spectrum
Our FPGAs need a stable 100MHz PCIE clock to work. The used FSP config
thing looked
void mainboard_memory_init_params(FSPM_UPD *mupd)
{
FSP_M_CONFIG *mem_cfg;
struct spd_block blk = {
.addr_map = { 0x50 },
};
mem_cfg = &mupd->FspmConfig;
mem_cfg->PegDisableSpreadSpectrumClocking = 1;
mem_cfg->PchPmPciePllSsc = 0;
...
}
With this configuration the PCIe reference clock was off more then 8% which
caused the system to hang during cold and warm boots.
In the next step I removed assignment of PchPmPciePllSsc as it is documented
as 'No BIOS override'. With this change I got more then 1000 soft and
2000 hard reboots
without any problem. Keep in mind we started with only 10 successful reboots.

Please be more specific about the final setting of this UPD. `No BIOS
override` is the documentation for the default value of 0xff. But is
this set to the default in the binary? who knows...

void mainboard_memory_init_params(FSPM_UPD *mupd)
{
FSP_M_CONFIG *mem_cfg;
struct spd_block blk = {
.addr_map = { 0x50 },
};
mem_cfg = &mupd->FspmConfig;
/* Disable PCIe Spread Spectrum Clocking */
printk(BIOS_ERR, "PegDisableSpreadSpectrumClocking: %x\n",
mem_cfg->PegDisableSpreadSpectrumClocking);
printk(BIOS_ERR, "PchPmPciePllSsc: %x\n", mem_cfg->PchPmPciePllSsc);
mem_cfg->PegDisableSpreadSpectrumClocking = 1;
get_spd_smbus(&blk);
dump_spd_info(&blk);
assert(blk.spd_array[0][0] != 0);
mainboard_fill_dq_map_data(&mem_cfg->DqByteMapCh0);
mainboard_fill_dqs_map_data(&mem_cfg->DqsMapCpu2DramCh0);
mainboard_fill_rcomp_res_data(&mem_cfg->RcompResistor);
mainboard_fill_rcomp_strength_data(&mem_cfg->RcompTarget);
mem_cfg->DqPinsInterleaved = TRUE;
mem_cfg->MemorySpdDataLen = blk.len;
mem_cfg->MemorySpdPtr00 = (uintptr_t) blk.spd_array[0];
}
..
FMAP: base = ff000000 size = 1000000 #areas = 4
MRC: no data in 'RW_MRC_CACHE'
PegDisableSpreadSpectrumClocking: 0
PchPmPciePllSsc: ff
SPD: module type is DDR4
..

Post by Nico Huber

Post by Christian Gmeiner
The big problem is that PegDisableSpreadSpectrumClocking has no effect
at all. I measured
the freq it is not the 100MHz as expected. And I need to have a stable
100MHz this clock source
is used internally by the FPGA to drive internal clocks. The end
results is that EtherCAT is not
able to sync.

This setting is about a different clock, I guess. Can you please clarify
what is connected to which clock on your board.

Our FPGAs are using the 100 MHz PCIe clock as input to drive internal
clocks etc. One
of these clock is used for EtherCAT. If spread spectrum is active we
are around 100 MHz
(worst measured freq was ~92 MHz) and as a result the internal FPGA clock is not
stable/reliable --> EtherCAT sync fails.
mem_cfg->PegDisableSpreadSpectrumClocking = 1;
mem_cfg->PchPmPciePllSsc = 0;
I get a stable 100 MHz PCIe clock signal and everything works - except
the device hangs
after < 10 warm reboots. Looks like PCIe link training fails
uncountable times (seen with protocol
analyzer).
--
greets
--
Christian Gmeiner, MSc
https://christian-gmeiner.info
--
https://mail.coreboot.org/mailman/listinfo/coreboot

--
Best regards,
Naresh G. Solanki

--
greets
--
Christian Gmeiner, MSc

https://christian-gmeiner.info

[Peter Stuge]

Post by Christian Gmeiner
The system does not hang if I only change
mem_cfg->PegDisableSpreadSpectrumClocking = 1;
But it has no effect on the PCIe reference clock and it looks like
spread spectrum is still used.

AFAIK Peg refers to PCI Express Graphics. Maybe there's another,
per-port, setting?


//Peter

[Christian Gmeiner]

Post by Peter Stuge

Post by Christian Gmeiner
The system does not hang if I only change
mem_cfg->PegDisableSpreadSpectrumClocking = 1;
But it has no effect on the PCIe reference clock and it looks like
spread spectrum is still used.

AFAIK Peg refers to PCI Express Graphics. Maybe there's another,
per-port, setting?

Could be but and the comment is about this variable is wrong:
https://github.com/coreboot/coreboot/blob/f3122426b851b9ca009e501a8d8c60d062f84c98/src/vendorcode/intel/fsp/fsp2_0/skykabylake/FspmUpd.h#L534
--
greets
--
Christian Gmeiner, MSc

https://christian-gmeiner.info

[Naresh G. Solanki]

Hi,

A1. PLTRST# not getting de-asserted might be a separate issue. I
recommend you debug power on sequence & find the exact point of
failures like powerok, VR stability(check for over/undervoltage) near
SoC.
A2. Do same VR's provide power to FPGA & there is no power shortage ?

B1. SoC is clocked by external crystal, Can you please check Crystal
OSC frequency, accuracy/deviation from expected values. I'm guessing
that the external crystal frequency is around 8% lower than expected
levels.
B2. If possible provide waveform of external crystal & clksrc

Regards,
Naresh G Solanki
On Tue, Oct 16, 2018 at 3:13 PM Christian Gmeiner

Post by Christian Gmeiner

Post by Peter Stuge

Post by Christian Gmeiner
The system does not hang if I only change
mem_cfg->PegDisableSpreadSpectrumClocking = 1;
But it has no effect on the PCIe reference clock and it looks like
spread spectrum is still used.

AFAIK Peg refers to PCI Express Graphics. Maybe there's another,
per-port, setting?

https://github.com/coreboot/coreboot/blob/f3122426b851b9ca009e501a8d8c60d062f84c98/src/vendorcode/intel/fsp/fsp2_0/skykabylake/FspmUpd.h#L534
--
greets
--
Christian Gmeiner, MSc
https://christian-gmeiner.info
--
https://mail.coreboot.org/mailman/listinfo/coreboot

[Nico Huber]

Post by Christian Gmeiner

Post by Peter Stuge

Post by Christian Gmeiner
The system does not hang if I only change
mem_cfg->PegDisableSpreadSpectrumClocking = 1;
But it has no effect on the PCIe reference clock and it looks like
spread spectrum is still used.

AFAIK Peg refers to PCI Express Graphics. Maybe there's another,
per-port, setting?

https://github.com/coreboot/coreboot/blob/f3122426b851b9ca009e501a8d8c60d062f84c98/src/vendorcode/intel/fsp/fsp2_0/skykabylake/FspmUpd.h#L534

Why wrong? It just doesn't tell for what clock source it is. Or clock
input? even that's not clear to me. But I agree that it doesn't seem
to apply in your case. PEG is PCIe too, but not a feature of your
SKU, AFAICT.

Nico

[Nico Huber]

Post by Christian Gmeiner
Am Mo., 15. Okt. 2018 um 15:26 Uhr schrieb Naresh G. Solanki

Post by Naresh G. Solanki
For problem 1, can you give data on where exactly it hangs?, Is it in
OS or FW ?, Can you provide kernel/coreboot log, port 80 dump when it
hangs.

mem_cfg->PegDisableSpreadSpectrumClocking = 1;
mem_cfg->PchPmPciePllSsc = 0;
The 'physical' cause for the hang is also known: PLT_RST# never gets
high again. There are chances
that it does not hang but PCI devices (sata, usb hc, ...) are not
working as expected as the PCe reference clock
is in such a case at around 92 Mhz.

You said earlier that you can dump (but not change) the ICC settings,
right? Did you check what settings are made with PchPmPciePllSsc == 0?

If FSP ends up enabling it with a value of 0, I could imagine that this
is just unexpected / the result undefined. Did you try other values than
0? 1? I guess the value gives a percentage of a percentage; 0.01% down-
spread is probably not too bad.

And maybe somebody with access to the FSP source can document what it
does and if there is a switch (missing) to disabled it ;)

Nico