<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style></head><body lang=PL link=blue vlink="#954F72"><div class=WordSection1><p class=MsoNormal>Frederic,</p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I forgot abot important information – yesterday I pushed DT alignment to v4.16-rc4 on top of narvell-armada-wip. Please use it and let know how it works in your setup.</p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Thanks,</p><p class=MsoNormal>Marcin</p><p class=MsoNormal><o:p> </o:p></p><div style='mso-element:para-border-div;border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal style='border:none;padding:0cm'><b>Od: </b><a href="mailto:mw@semihalf.com">Marcin Wojtas</a><br><b>Wysłano: </b>wtorek, 13 marca 2018 07:48<br><b>Do: </b><a href="mailto:frederik.lotter@netronome.com">Frederik Lotter</a><br><b>DW: </b><a href="mailto:macchiato@lists.einval.com">macchiato@lists.einval.com</a>; <a href="mailto:ard.biesheuvel@linaro.org">Ard Biesheuvel</a><br><b>Temat: </b>Re: [Macchiato] EDKII grub boot fails with PCIe init</p></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>2018-03-13 6:49 GMT+01:00 Frederik Lotter <frederik.lotter@netronome.com>:</p><p class=MsoNormal>> On 12 Mar 2018 6:45 PM, "Marcin Wojtas" <mw@semihalf.com> wrote:</p><p class=MsoNormal>><o:p> </o:p></p><p class=MsoNormal>> Yes, it's the latest - I've just pushed a fix for OS variables write,</p><p class=MsoNormal>> so you can fetch. I have a couple of questions:</p><p class=MsoNormal>><o:p> </o:p></p><p class=MsoNormal>> 0. What version of the board are you using (is the PCIE-reset fix there)?</p><p class=MsoNormal>><o:p> </o:p></p><p class=MsoNormal>><o:p> </o:p></p><p class=MsoNormal>> We have v1.3 revision boards.</p><p class=MsoNormal>><o:p> </o:p></p><p class=MsoNormal>><o:p> </o:p></p><p class=MsoNormal>> 1. What kind of pcie device are you using?</p><p class=MsoNormal>><o:p> </o:p></p><p class=MsoNormal>><o:p> </o:p></p><p class=MsoNormal>> We have Smart NIC cards, but this board has nothing plugged in.</p><p class=MsoNormal>><o:p> </o:p></p><p class=MsoNormal>><o:p> </o:p></p><p class=MsoNormal>> 2. Does the boot from u-boot always succeed with the pcie device?</p><p class=MsoNormal>><o:p> </o:p></p><p class=MsoNormal>><o:p> </o:p></p><p class=MsoNormal>> Yes. It appears 100% reliable in comparison with EFI boot.</p><p class=MsoNormal>><o:p> </o:p></p><p class=MsoNormal>> So my conclusion is either the EDK Phy setup is different, or the DT data is</p><p class=MsoNormal>> somehow different.</p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Phy setup should be aligned, soon it will be common fortunately. DT is</p><p class=MsoNormal>a bit different as it uses 512MB MMIO32 region, but his should not</p><p class=MsoNormal>have any impact. Overall this is strange, I've booted kernel hundreds</p><p class=MsoNormal>of times without issues. However I usually had at least e1000 NIC</p><p class=MsoNormal>plugged in my setup (now also x4 GPU card on the second board). So I</p><p class=MsoNormal>have a couple of requests, so that I can better understand the issue:</p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>1. Can you provide me with full boot log (beginning from first lines</p><p class=MsoNormal>after power-on/reset) from both u-boot and uefi?</p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>2. Can you plug the smart nic and try efi boot to see if the problem persists?</p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>3. Can you try cross-checking the DT</p><p class=MsoNormal>- take armada-8040-mcbin.dtb from Platforms/Marvell/Armada and boot</p><p class=MsoNormal>your linux from u-boot?</p><p class=MsoNormal>- take the DT from u-boot and substitute above file in uefi, rebuild</p><p class=MsoNormal>your image and boot?</p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Thanks,</p><p class=MsoNormal>Marcin</p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>><o:p> </o:p></p><p class=MsoNormal>><o:p> </o:p></p><p class=MsoNormal>> Thanks,</p><p class=MsoNormal>> Marcin</p><p class=MsoNormal>><o:p> </o:p></p><p class=MsoNormal>><o:p> </o:p></p><p class=MsoNormal>> 2018-03-12 17:31 GMT+01:00 Frederik Lotter <frederik.lotter@netronome.com>:</p><p class=MsoNormal>>> 150e1b404d2e3b49574ad57e485827be12270ab9</p><p class=MsoNormal>>><o:p> </o:p></p><p class=MsoNormal>>> I believe this is HEAD, committed on 6th of March?</p><p class=MsoNormal>>><o:p> </o:p></p><p class=MsoNormal>>> On Mon, Mar 12, 2018 at 6:23 PM, Marcin Wojtas <mw@semihalf.com> wrote:</p><p class=MsoNormal>>>><o:p> </o:p></p><p class=MsoNormal>>>> Hi Frederik,</p><p class=MsoNormal>>>><o:p> </o:p></p><p class=MsoNormal>>>> Which commit ID of marvell-armada-wip you're building your binary from?</p><p class=MsoNormal>>>><o:p> </o:p></p><p class=MsoNormal>>>> Thanks,</p><p class=MsoNormal>>>> Marcin</p><p class=MsoNormal>>>><o:p> </o:p></p><p class=MsoNormal>>>> 2018-03-12 17:22 GMT+01:00 Ard Biesheuvel <ard.biesheuvel@linaro.org>:</p><p class=MsoNormal>>>> > On 12 March 2018 at 16:15, Frederik Lotter</p><p class=MsoNormal>>>> > <frederik.lotter@netronome.com> wrote:</p><p class=MsoNormal>>>> >> Hi,</p><p class=MsoNormal>>>> >> I am getting CPU stall warnings when booting up using the EFI route. I</p><p class=MsoNormal>>>> >> suspect the PCIe interface, as the stall warning sometimes contain the</p><p class=MsoNormal>>>> >> probe</p><p class=MsoNormal>>>> >> function. Other times is seems to get further than PCIe init, but</p><p class=MsoNormal>>>> >> still</p><p class=MsoNormal>>>> >> stall interrupt handling.</p><p class=MsoNormal>>>> >> Here are some facts around my observation:</p><p class=MsoNormal>>>> >></p><p class=MsoNormal>>>> >> I have two sdcards for my Machiattobin board. They have identical</p><p class=MsoNormal>>>> >> kernels</p><p class=MsoNormal>>>> >> (4.16 rc5) with Ubuntu 16.04 rootfs. The one sdcard uses a uboot, DT</p><p class=MsoNormal>>>> >> and</p><p class=MsoNormal>>>> >> kernel boot. The second sdcard has EDKII, grub kernel boot. The EDKII</p><p class=MsoNormal>>>> >> build</p><p class=MsoNormal>>>> >> includes the device tree DTB (and DTS which I believe is unused) from</p><p class=MsoNormal>>>> >> the</p><p class=MsoNormal>>>> >> one used on the uboot sdcard.</p><p class=MsoNormal>>>> >></p><p class=MsoNormal>>>> >> EFI stub: Booting Linux Kernel...</p><p class=MsoNormal>>>> >> EFI stub: Using DTB from configuration table</p><p class=MsoNormal>>>> >> EFI stub: Exiting boot services and installing virtual address map...</p><p class=MsoNormal>>>> >> [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd081]</p><p class=MsoNormal>>>> >> [ 0.000000] Linux version 4.16.0-rc5-mbcin-netronome-2-dirty</p><p class=MsoNormal>>>> >> (root@mcb1-cpt) (gcc version 5.4.0 20160609 (Ubuntu/Linaro</p><p class=MsoNormal>>>> >> 5.4.0-6ubuntu1~16.04.9)) #2 SMP PREEMPT Mon Mar 12 14:40:25 UTC 2018</p><p class=MsoNormal>>>> >> [ 0.000000] Machine model: Marvell 8040 MACHIATOBin</p><p class=MsoNormal>>>> >> [ 0.000000] efi: Getting EFI parameters from FDT:</p><p class=MsoNormal>>>> >> [ 0.000000] efi: EFI v2.70 by EDK II</p><p class=MsoNormal>>>> >> [ 0.000000] efi: SMBIOS 3.0=0xbfd00000 ACPI 2.0=0xb6760000</p><p class=MsoNormal>>>> >> MEMATTR=0xb8973418 RNG=0xbffdbf98</p><p class=MsoNormal>>>> >> [ 0.000000] random: fast init done</p><p class=MsoNormal>>>> >> [ 0.000000] efi: seeding entropy pool</p><p class=MsoNormal>>>> >> :</p><p class=MsoNormal>>>> >></p><p class=MsoNormal>>>> >> (I am using the latest EDKII master, the Marvell edk2-open-platform</p><p class=MsoNormal>>>> >> 17.10</p><p class=MsoNormal>>>> >> banch, with all the latest mv-ddr/ atf /etc....).</p><p class=MsoNormal>>>> >></p><p class=MsoNormal>>>> >> The DT data appear there in die EFI boot, but the PCIe interface</p><p class=MsoNormal>>>> >> fails,</p><p class=MsoNormal>>>> >> and</p><p class=MsoNormal>>>> >> results (I believe) in the CPU stall warnings:</p><p class=MsoNormal>>>> >></p><p class=MsoNormal>>>> >> [ 717.453025] INFO: rcu_preempt self-detected stall on CPU</p><p class=MsoNormal>>>> >> :</p><p class=MsoNormal>>>> >> :</p><p class=MsoNormal>>>> >> [ 717.589783] armada8k_pcie_probe+0x140/0x240</p><p class=MsoNormal>>>> >> :</p><p class=MsoNormal>>>> >></p><p class=MsoNormal>>>> >> Other times, the pcie gets further:</p><p class=MsoNormal>>>> >></p><p class=MsoNormal>>>> >> [ 3.312127] PCI: OF: host bridge /cp0/pcie@f2600000 ranges:</p><p class=MsoNormal>>>> >> [ 3.317740] PCI: OF: IO 0xf9000000..0xf900ffff -> 0xf9000000</p><p class=MsoNormal>>>> >> [ 3.323692] PCI: OF: MEM 0xc0000000..0xdfffffff -> 0xc0000000</p><p class=MsoNormal>>>> >> [ 3.328915] random: crng init done</p><p class=MsoNormal>>>> >> [ 4.326158] armada8k-pcie f2600000.pcie: phy link never came up</p><p class=MsoNormal>>>> >> [ 4.332109] armada8k-pcie f2600000.pcie: Link not up after</p><p class=MsoNormal>>>> >> reconfiguration</p><p class=MsoNormal>>>> >> [ 4.339056] armada8k-pcie f2600000.pcie: PCI host bridge to bus</p><p class=MsoNormal>>>> >> 0000:00</p><p class=MsoNormal>>>> ></p><p class=MsoNormal>>>> ></p><p class=MsoNormal>>>> > To be brutally honest, the armada8k-pcie driver is a piece of junk,</p><p class=MsoNormal>>>> > and you're much better off using the generic ECAM driver, which now</p><p class=MsoNormal>>>> > includes special handling for the missing root port on Synopsys IP.</p><p class=MsoNormal>>>> ></p><p class=MsoNormal>>>> > It also allows you to have both MMIO32 and MMIO64 regions, which can</p><p class=MsoNormal>>>> > be useful with some PCIe cards with large BARs</p><p class=MsoNormal>>>> ></p><p class=MsoNormal>>>> > Could you try</p><p class=MsoNormal>>>> ></p><p class=MsoNormal>>>> > compatible = "marvell,armada8k-pcie-ecam";</p><p class=MsoNormal>>>> ></p><p class=MsoNormal>>>> > in the DT node, please?</p><p class=MsoNormal>>>> ></p><p class=MsoNormal>>>> > (Before you do that, please check whether UEFI recognizes your PCI</p><p class=MsoNormal>>>> > hardware using the 'pci' command in the shell)</p><p class=MsoNormal>>>> ></p><p class=MsoNormal>>>> ></p><p class=MsoNormal>>>> >> [ 4.345705] pci_bus 0000:00: root bus resource [bus 00-ff]</p><p class=MsoNormal>>>> >> [ 4.351217] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]</p><p class=MsoNormal>>>> >> (bus</p><p class=MsoNormal>>>> >> address [0xf9000000-0xf900ffff])</p><p class=MsoNormal>>>> >> [ 4.360741] pci_bus 0000:00: root bus resource [mem</p><p class=MsoNormal>>>> >> 0xc0000000-0xdfffffff]</p><p class=MsoNormal>>>> >> [ 4.367659] pci 0000:00:00.0: [11ab:0110] type 01 class 0x060400</p><p class=MsoNormal>>>> >> [ 4.373708] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x000fffff</p><p class=MsoNormal>>>> >> 64bit]</p><p class=MsoNormal>>>> >> [ 4.380562] pci 0000:00:00.0: supports D1 D2</p><p class=MsoNormal>>>> >> [ 4.384853] pci 0000:00:00.0: PME# supported from D0 D1 D3hot</p><p class=MsoNormal>>>> >> [ 4.390697] pci 0000:00:00.0: bridge configuration invalid ([bus</p><p class=MsoNormal>>>> >> 00-00]),</p><p class=MsoNormal>>>> >> reconfiguring</p><p class=MsoNormal>>>> >> [ 4.398771] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated</p><p class=MsoNormal>>>> >> to</p><p class=MsoNormal>>>> >> 01</p><p class=MsoNormal>>>> >> [ 4.405427] pci 0000:00:00.0: BAR 0: assigned [mem</p><p class=MsoNormal>>>> >> 0xc0000000-0xc00fffff</p><p class=MsoNormal>>>> >> 64bit]</p><p class=MsoNormal>>>> >> [ 4.412776] pci 0000:00:00.0: PCI bridge to [bus 01]</p><p class=MsoNormal>>>> >> [ 4.725111] pcieport 0000:00:00.0: Signaling PME with IRQ 56</p><p class=MsoNormal>>>> >> [ 4.730842] pcieport 0000:00:00.0: AER enabled with IRQ 56</p><p class=MsoNormal>>>> >></p><p class=MsoNormal>>>> >> but then CPUs are still stalled on some incoming IRQ</p><p class=MsoNormal>>>> >></p><p class=MsoNormal>>>> >> [ 87.352768] dump_backtrace+0x0/0x150</p><p class=MsoNormal>>>> >> [ 87.356445] show_stack+0x14/0x20</p><p class=MsoNormal>>>> >> [ 87.359773] sched_show_task+0x14c/0x170</p><p class=MsoNormal>>>> >> [ 87.363711] dump_cpu_task+0x40/0x50</p><p class=MsoNormal>>>> >> [ 87.367300] rcu_dump_cpu_stacks+0x94/0xd8</p><p class=MsoNormal>>>> >> [ 87.371412] rcu_check_callbacks+0x7ac/0x980</p><p class=MsoNormal>>>> >> [ 87.375700] update_process_times+0x2c/0x58</p><p class=MsoNormal>>>> >> [ 87.379900] tick_sched_handle.isra.5+0x30/0x50</p><p class=MsoNormal>>>> >> [ 87.384449] tick_sched_timer+0x40/0x90</p><p class=MsoNormal>>>> >> [ 87.388301] __hrtimer_run_queues+0x124/0x198</p><p class=MsoNormal>>>> >> [ 87.392676] hrtimer_interrupt+0xe4/0x240</p><p class=MsoNormal>>>> >> [ 87.396701] arch_timer_handler_phys+0x30/0x40</p><p class=MsoNormal>>>> >> [ 87.401163] handle_percpu_devid_irq+0x78/0x130</p><p class=MsoNormal>>>> >> [ 87.405712] generic_handle_irq+0x24/0x38</p><p class=MsoNormal>>>> >> [ 87.409738] __handle_domain_irq+0x5c/0xb8</p><p class=MsoNormal>>>> >> [ 87.413850] gic_handle_irq+0x58/0xb0</p><p class=MsoNormal>>>> >> [ 87.417526] el1_irq+0xb0/0x128</p><p class=MsoNormal>>>> >> [ 87.420678] __do_softirq+0xb0/0x228</p><p class=MsoNormal>>>> >> [ 87.424267] irq_exit+0xbc/0xf0</p><p class=MsoNormal>>>> >> [ 87.427421] __handle_domain_irq+0x60/0xb8</p><p class=MsoNormal>>>> >> [ 87.431533] gic_handle_irq+0x58/0xb0</p><p class=MsoNormal>>>> >> [ 87.435209] el1_irq+0xb0/0x128</p><p class=MsoNormal>>>> >></p><p class=MsoNormal>>>> >> Is anyone aware of any issue like this?</p><p class=MsoNormal>>>> >></p><p class=MsoNormal>>>> >> Regards,</p><p class=MsoNormal>>>> >> Fred</p><p class=MsoNormal>>>> >></p><p class=MsoNormal>>>> >></p><p class=MsoNormal>>>> >> _______________________________________________</p><p class=MsoNormal>>>> >> Macchiato mailing list</p><p class=MsoNormal>>>> >> Macchiato@lists.einval.com</p><p class=MsoNormal>>>> >> https://lists.einval.com/cgi-bin/mailman/listinfo/macchiato</p><p class=MsoNormal>>>> ></p><p class=MsoNormal>>>> > _______________________________________________</p><p class=MsoNormal>>>> > Macchiato mailing list</p><p class=MsoNormal>>>> > Macchiato@lists.einval.com</p><p class=MsoNormal>>>> > https://lists.einval.com/cgi-bin/mailman/listinfo/macchiato</p><p class=MsoNormal>>><o:p> </o:p></p><p class=MsoNormal>>><o:p> </o:p></p><p class=MsoNormal>><o:p> </o:p></p><p class=MsoNormal>><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p></div></body></html>