[Macchiato] MacchiatoBin GPU progress [#freetimeproject]

[Ard Biesheuvel]

On 5 June 2017 at 12:01, Matt Spencer <Matt.Spencer at arm.com> wrote:
> Hey Ard
>
>
> Just wanted to touch base on this issue - has there been any progress beyond
> the patches you suggested in your last mail?  Are we any closer to an
> official solution?  And is there something that I am able to try out on my
> hardware yet?
>

Hello Matt,

I managed to depopulate the JTAG header on my board, and gently hack
away the back of the PCIe connector, so I can plug x8 and x86 cards,
and eliminate the riser from the equation.

As it turns out, this does not make a whole lot of difference: I have
a x1 Realtek 8169 NIC that is rock solid, a couple of x1 XHCI cards
that are so so, and beyond that, no x8 or x16 cards can train
successfully at all. This strengthens my suspicion that there are
either signal integrity or power routing issues with this board.

Does your GPU card's link train successfully every time? And does your
riser have a separate power lead connected?

I have asked Semihalf and Marvell for feedback regarding the types and
brands of PCIe cards that they tested, but given the insufficient size
of the PCIe MMIO windows, it is pretty obvious that they never tried
any graphics in the first place.

We (as Linaro, or ARM for that matter) are not really in a position to
debug this. These issues may be corrected with increased drive
strength settings, or other board integration level controls on the
SoC that we have no documentation for, nor the means to discover the
correct values. So the ball is really in Marvell's court on this one,
I'm afraid.

-- 
Ard.

> ________________________________
> From: Ard Biesheuvel <ard.biesheuvel at linaro.org>
> Sent: 06 May 2017 09:14:29
> To: Matt Spencer
> Cc: Leif Lindholm; Wookey; Steve Capper; Steve McIntyre;
> macchiato at lists.einval.com
> Subject: Re: MacchiatoBin GPU progress [#freetimeproject]
>
> On 5 May 2017 at 18:02, Matt Spencer <Matt.Spencer at arm.com> wrote:
>> Hi Ard, Leif has suggested I add you to this thread.
>>
>>
>> I am trying to get the MacchiatoBin working with a PCIe GPU.  The approach
>> I
>> am taking is to use a powered 16->1 lane PCIe adapter
>>
>> (https://www.amazon.co.uk/gp/product/B01ER2Z1GY/ref=oh_aui_detailpage_o03_s00?ie=UTF8&psc=1)
>> and using a modern NVidia GForce GPU that is know to function well with
>> Nouveau
>>
>> (https://www.amazon.co.uk/gp/product/B01AY7927A/ref=oh_aui_detailpage_o04_s00?ie=UTF8&psc=1).
>>
>>
>> I have the kernel booting, detecting the graphics card and trying to
>> initialise the driver, but come across this issue:
>>
>
> ...
>> root at localhost:~# lspci
>>
>> 00:00.0 PCI bridge: Marvell Technology Group Ltd. Device 0110
>>
>> 01:00.0 VGA compatible controller: NVIDIA Corporation Device 128b (rev a1)
>>
>> 01:00.1 Audio device: NVIDIA Corporation GK208 HDMI/DP Audio Controller
>> (rev
>> a1)
>>
>>
>>
>> It looks like I don't have enough BAR memory available?  Leif tells me
>> that
>> the base configuration only exposes 15M, but you have a different firmware
>> that could solve the problem?
>>
>
> Correct. The firmware configuration of this board is slightly silly,
> but can be improved a lot by reconfiguring the translation windows at
> various levels to allow for a much larger BAR space (512 MB + 4GB
> rather than 15 MB, which is simply insufficient for graphics cards)
>
> The problem is that, while these changes could easily be applied to
> the Marvell version of u-boot, we currently only have a firmware stack
> based on UEFI and the upstream version of the device tree (rather than
> the wildly incompatible Marvell one), and there are other issues we
> need to sort out before this stack is usable enough (i.e., you will
> lose your network connection if you switch to it now)
>
> As for the stability issues we identified: our goal is to produce a
> firmware stack for this board that allows you to boot bog standard
> distro installers, whose kernels trigger this issue, and currently,
> the only workaround that is guaranteed to prevent this issue (even for
> KVM guests) is to take down two of the four cores in the firmware.
> Graphics card support is high on our list as well, but it will simply
> take us some time to get all of this sorted.
>
> In the mean time, you are welcome to have a look at the required
> changes on the ATF side: this patch
>
> https://github.com/ardbiesheuvel/arm-trusted-firmware/commit/591d1cb7c3bd4fa807093643ef8300d15973818e
>
> is the only one you will need for ATF. Next, you will need to add the
> window 0xc0000000-0xdfffffff to the PCIe DT node, which should be
> sufficient to get the driver to allocate BARs from it. However, given
> that this range was formerly in use for DRAM, the DT description of
> memory needs to be updated as well, and this slice removed. (In the
> UEFI code, I actually remap it so you don't lose it, but just hiding
> it from the kernel should be sufficient in the short term)
>
> Hope this helps,
> Ard.
> IMPORTANT NOTICE: The contents of this email and any attachments are
> confidential and may also be privileged. If you are not the intended
> recipient, please notify the sender immediately and do not disclose the
> contents to any other person, use it for any purpose, or store or copy the
> information in any medium. Thank you.