<div dir="ltr">Hi,<div class="gmail_extra"><br><div class="gmail_quote">On Wed, Apr 25, 2018 at 9:02 AM, Frederik Lotter <span dir="ltr"><<a href="mailto:frederik.lotter@netronome.com" target="_blank">frederik.lotter@netronome.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span class="gmail-">On Tue, Apr 24, 2018 at 6:07 PM, Marcin Wojtas <span dir="ltr"><<a href="mailto:mw@semihalf.com" target="_blank">mw@semihalf.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Frederik,<div><br></div><div>ARM-TF is ready to support extended PCIE windows @0xc0000000 and @0x800000000. Maybe you are using some old version of it? Can you show the bootlog from the very beginning?</div></div></blockquote><div><br></div></span><div><div>BootROM - 2.03</div><div>Starting CP-0 IOROM 1.07</div><div>Booting from SD 0 (0x29)</div><div>Found valid image at boot postion 0x002</div><div>lNOTICE: Starting binary extension</div><div>NOTICE: SVC: SW Revision 0x0. SVC is not supported</div><div>mv_ddr: mv_ddr-armada-17.10.4-gcc280f0 (Apr 25 2018 - 06:57:16)</div><div>mv_ddr: completed successfully</div><div>NOTICE: Cold boot</div><div>NOTICE: Booting Trusted Firmware</div><div>NOTICE: BL1: v1.3(release):armada-17.10.7:<wbr>4396548</div><div>NOTICE: BL1: Built : 06:57:31, Apr 25 2018</div><div>NOTICE: BL1: Booting BL2</div><div>lNOTICE: BL2: v1.3(release):armada-17.10.7:<wbr>4396548</div><div>NOTICE: BL2: Built : 06:57:34, Apr 25 2018</div><div>BL2: Initiating SCP_BL2 transfer to SCP</div><div>NOTICE: SCP_BL2 contains 2 concatenated images</div><div>NOTICE: Load image to CP1 MSS</div><div>NOTICE: Loading MSS image from address 0x4023020 Size 0x1378 to MSS at 0xf4280000</div><div>NOTICE: Done</div><div>NOTICE: Load image to AP MSS</div><div>NOTICE: Loading MSS image from address 0x4024398 Size 0x1f68 to MSS at 0xf0580000</div><div>N</div><div><br></div><div>FreeRTOS 7.3.0 - Marvell cm3 - A8K release armada-17.10.1</div><div><br></div><div>OTICE: Done</div><div>NOTICE: SCP Image doesn't contain PM firmware</div><div>NOTICE: BL1: Booting BL31</div><div>lNOTICE: MSS PM is not supported in this build</div><div>NOTICE: BL31: v1.3(release):armada-17.10.7:<wbr>4396548</div><div>NOTICE: BL31: Built : 06:57:41, Apr 25 2018</div><div>l</div><div><br></div><div>U-Boot 2017.03-armada-17.10.2-<wbr>g6a6581a-dirty (Apr 25 2018 - 06:56:38 +0000)</div><div><br></div><div>Model: MACCHIATOBin-8040</div><div>Clock: CPU 2000 [MHz]</div><div> DDR 1200 [MHz]</div><div> FABRIC 1200 [MHz]</div><div> MSS 200 [MHz]</div><div>DRAM: 4 GiB</div><div>U-Boot DT blob at : 000000007f70ec38</div><div>EEPROM configuration pattern not detected.</div><div>Comphy chip #0:</div><div>Comphy-0: PEX0</div><div>Comphy-1: PEX0</div><div>Comphy-2: PEX0</div><div>Comphy-3: PEX0</div><div>Comphy-4: SFI</div><div>Comphy-5: SATA1</div><div>Comphy chip #1:</div><div>Comphy-0: SGMII1 1.25 Gbps</div><div>Comphy-1: SATA0</div><div>Comphy-2: USB3_HOST0</div><div>Comphy-3: SATA1</div><div>Comphy-4: SFI</div><div>Comphy-5: SGMII2 3.125 Gbps</div><div>UTMI PHY 0 initialized to USB Host0</div><div>SATA link 0 timeout.</div><div>SATA link 1 timeout.</div><div>AHCI 0001.0000 32 slots 2 ports 6 Gbps 0x3 impl SATA mode</div><div>flags: 64bit ncq led only pmp fbss pio slum part sxs</div><div>SATA link 0 timeout.</div><div>going to start mss_periodic_ihb_wa for mss</div><div>SATA link 1 timeout.</div><div>AHCI 0001.0000 32 slots 2 ports 6 Gbps 0x3 impl SATA mode</div><div>flags: 64bit ncq led only pmp fbss pio slum part sxs</div><div>pci_uclass_pre_probe, bus=0/pcie@f2600000, parent=cp110-master</div><div>decode_regions: len=14, cells_per_record=7</div><div>decode_regions: region 0, pci_addr=f9000000, addr=f9000000, size=10000, space_code=1</div><div> - type=1, pos=0</div><div>decode_regions: region 1, pci_addr=c0000000, addr=c0000000, size=20000000, space_code=2</div><div> - type=0, pos=1</div><div>PCIE-0: Link up (Gen2-x4, Bus0)</div><div>pci_uclass_post_probe: probing bus 0</div><div>pci_bind_bus_devices: bus 0/pcie@f2600000: found device 0, function 0</div><div>pci_find_and_bind_driver: Searching for driver: vendor=11ab, device=110</div><div>pci_find_and_bind_driver: No match found: bound generic driver instead</div><div>pci_auto_config_devices: start</div><div>PCI Autoconfig: Bus Memory region: [c0000000-dfffffff],</div><div> Physical Memory [c0000000-dfffffffx]</div><div>PCI Autoconfig: Bus I/O region: [f9000000-f900ffff],</div><div> Physical Memory [f9000000-f900ffffx]</div><div>pci_auto_config_devices: device pci_0:0.0</div><div>PCI Autoconfig: Found P2P bridge, device 0</div><div>PCI Autoconfig: BAR 0, I/O, size=0xfff4, address=0xf9000000 bus_lower=0xf900fff4</div><div>dm_pci_hose_probe_bus</div><div>pci_get_bus_max: ret=0</div><div>dm_pci_hose_probe_bus: bus = 1/pci_0:0.0</div><div>pci_uclass_pre_probe, bus=1/pci_0:0.0, parent=pcie@f2600000</div><div>pci_uclass_post_probe: probing bus 1</div><div>pci_bind_bus_devices: bus 1/pci_0:0.0: found device 0, function 0</div><div>pci_find_and_bind_driver: Searching for driver: vendor=8086, device=1521</div><div>pci_find_and_bind_driver: No match found: bound generic driver instead</div><div>pci_bind_bus_devices: bus 1/pci_0:0.0: found device 0, function 1</div><div>pci_find_and_bind_driver: Searching for driver: vendor=8086, device=1521</div><div>pci_find_and_bind_driver: No match found: bound generic driver instead</div><div>pci_auto_config_devices: start</div><div>pci_auto_config_devices: device pci_1:0.0</div><div>PCI Autoconfig: BAR 0, Mem, size=0x100000, address=0xc0000000 bus_lower=0xc0100000</div><div>PCI Autoconfig: BAR 1, I/O, size=0x20, No room in resource</div><div>PCI Autoconfig: BAR 2, Mem, size=0x4000, address=0xc0100000 bus_lower=0xc0104000</div><div>PCI Autoconfig: ROM, size=0x80000, address=0xc0180000 bus_lower=0xc0200000</div><div>pci_auto_config_devices: device pci_1:0.1</div><div>PCI Autoconfig: BAR 0, Mem, size=0x100000, address=0xc0200000 bus_lower=0xc0300000</div><div>PCI Autoconfig: BAR 1, I/O, size=0x20, No room in resource</div><div>PCI Autoconfig: BAR 2, Mem, size=0x4000, address=0xc0300000 bus_lower=0xc0304000</div><div>PCI Autoconfig: ROM, size=0x80000, address=0xc0380000 bus_lower=0xc0400000</div><div>pci_auto_config_devices: done</div><div>pci_get_bus_max: ret=1</div><div>pci_auto_config_devices: done</div><div>MMC: sdhci@6e0000: 0, sdhci@780000: 1</div><div>Net: eth0: mvpp2-0 [PRIME]mdio_register: non unique device name 'ethernet@0'</div><div>, eth1: mvpp2-3, eth2: mvpp2-4, eth3: mvpp2-5</div><div>Hit any key to stop autoboot: 0</div><div>Marvell>></div><div>Marvell>></div><span class="gmail-"><div>Marvell>> pci 1</div><div>Scanning PCI devices on bus 1</div><div>BusDevFun VendorId DeviceId Device Class Sub-Class</div><div>______________________________<wbr>______________________________<wbr>_</div><div>01.00.00 0x8086 0x1521 Network controller 0x00</div><div>01.00.01 0x8086 0x1521 Network controller 0x00</div><div>Marvell>> pci bar 01.00.00</div><div>ID Base Size Width Type</div><div>------------------------------<wbr>----------------------------</div></span><div> 0 0x000000c0000000 0x00000000100000 32 MEM</div><div>Marvell>> </div><div>Marvell>> md.l 0x000000c0000000 10</div><div>c0000000:"Synchronous Abort" handler, esr 0x96000006</div><div>ELR: 7ff7da08</div><div>LR: 7ff7d9e0</div><div>x0 : 0000000000000009 x1 : 0000000000000000</div><div>x2 : 000000000000003a x3 : 00000000c0000000</div><div>x4 : 0000000000000000 x5 : 000000007ff9cbbe</div><div>x6 : 0000000000000004 x7 : 000000000000000f</div><div>x8 : 000000007f70e750 x9 : 0000000000000008</div><div>x10: 000000007f70e3b9 x11: 0000000000000021</div><div>x12: 0000000000000008 x13: 00000000ffffffff</div><div>x14: 000000007f70eb0c x15: 000000007f70ec38</div><div>x16: 000000007ff9b11b x17: 000000007ff971a6</div><div>x18: 000000007f715d18 x19: 0000000000000010</div><div>x20: 00000000c0000000 x21: 00000000c0000000</div><div>x22: 0000000000000004 x23: 0000000000000008</div><div>x24: 0000000000000009 x25: 0000000000000004</div><div>x26: 0000000000000004 x27: 000000007f70e918</div><div>x28: 0000000000000000 x29: 000000007f70e890</div><div><br></div><div>Resetting CPU ...</div><div><br></div><div>resetting ...</div></div></div></div></div></blockquote><div><br></div><div>So the issue was that u-boot needed the mapping ...</div><div><br></div><div><div>diff --git a/arch/arm/dts/armada-8040-mcbin.dts b/arch/arm/dts/armada-8040-mcbin.dts</div><div>index 67f23df..7bce8fb 100644</div><div>--- a/arch/arm/dts/armada-8040-mcbin.dts</div><div>+++ b/arch/arm/dts/armada-8040-mcbin.dts</div><div>@@ -118,6 +118,8 @@</div><div> /* PCIe x4 */</div><div> &cpm_pcie0 {</div><div> num-lanes = <4>;</div><div>+ ranges = <0x81000000 0 0xf9000000 0 0xf9000000 0 0x10000</div><div>+ 0x82000000 0 0xc0000000 0 0xc0000000 0 0x20000000>;</div><div> status = "okay";</div><div> };</div><div><br></div><div>diff --git a/arch/arm/mach-mvebu/armada8k/cpu.c b/arch/arm/mach-mvebu/armada8k/cpu.c</div><div>index a1a5a2a..fd66003 100644</div><div>--- a/arch/arm/mach-mvebu/armada8k/cpu.c</div><div>+++ b/arch/arm/mach-mvebu/armada8k/cpu.c</div><div>@@ -33,7 +33,7 @@ DECLARE_GLOBAL_DATA_PTR;</div><div> * can be easier removed later dynamically if an Armada 7k device is detected.</div><div> * For a detailed memory map, please see doc/mvebu/armada-8k-memory.txt</div><div> */</div><div>-#define ARMADA_7K8K_COMMON_REGIONS_START 2</div><div>+#define ARMADA_7K8K_COMMON_REGIONS_START 3</div><div> static struct mm_region mvebu_mem_map[] = {</div><div> /* Armada 80x0 memory regions include the CP1 (slave) units */</div><div> {</div><div>@@ -52,6 +52,14 @@ static struct mm_region mvebu_mem_map[] = {</div><div> .attrs = PTE_BLOCK_MEMTYPE(MT_DEVICE_NGNRNE) |</div><div> PTE_BLOCK_NON_SHARE</div><div> },</div><div>+ {</div><div>+ /* PCI CP0 regions */</div><div>+ .phys = 0xc0000000UL,</div><div>+ .virt = 0xc0000000UL,</div><div>+ .size = 0x20000000UL, /* 256MB */</div><div>+ .attrs = PTE_BLOCK_MEMTYPE(MT_DEVICE_NGNRNE) |</div><div>+ PTE_BLOCK_NON_SHARE</div><div>+ },</div><div> /* Armada 80x0 and 70x0 common memory regions start here */</div><div> {</div><div> /* RAM */</div></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><div class="gmail-h5"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div>Best regards,</div><div>Marcin</div></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="gmail-m_-8305540389335921814gmail-h5">2018-04-24 17:34 GMT+02:00 Ard Biesheuvel <span dir="ltr"><<a href="mailto:ard.biesheuvel@linaro.org" target="_blank">ard.biesheuvel@linaro.org</a>></span>:<br></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="gmail-m_-8305540389335921814gmail-h5">On 24 April 2018 at 17:32, Frederik Lotter<br>
<div><div class="gmail-m_-8305540389335921814gmail-m_3980863031430160498h5"><<a href="mailto:frederik.lotter@netronome.com" target="_blank">frederik.lotter@netronome.com</a><wbr>> wrote:<br>
> Hi,<br>
><br>
> The uboot build for mcbin get PCIe range data from the included DT.<br>
><br>
> ranges =<br>
> <0x81000000 0 0xf9000000 0 0xf9000000 0 0x10000<br>
> 0x82000000 0 0xf6000000 0 0xf6000000 0 0xf00000>;<br>
><br>
> Using uboot I can access BAR0 memory area inside uboot using a<br>
> standard Intel network card<br>
><br>
> Marvell>> pci 1<br>
> Scanning PCI devices on bus 1<br>
> BusDevFun VendorId DeviceId Device Class Sub-Class<br>
> ______________________________<wbr>______________________________<wbr>_<br>
> 01.00.00 0x8086 0x1521 Network controller 0x00<br>
> 01.00.01 0x8086 0x1521 Network controller 0x00<br>
> Marvell>> pci bar 01.00.00<br>
> ID Base Size Width Type<br>
> ------------------------------<wbr>----------------------------<br>
> 0 0x000000f6000000 0x00000000100000 32 MEM<br>
><br>
> Marvell>> md.l 0x000000f6000000 10<br>
> f6000000: 081c0241 081c0241 80280780 00000000 A...A.....(.....<br>
> f6000010: 0008471e 00000026 001400c0 0000004e .G..&.......N...<br>
> f6000020: 1411000a 00000000 00c28001 00000100 ................<br>
> f6000030: 00008808 00000000 81008100 00000000 ................<br>
><br>
> However, for our network card we need a much bigger BAR size, so<br>
> out DT entry is (which I took from the Linux DT which works):<br>
><br>
> ranges =<br>
> <0x81000000 0 0xf9000000 0 0xf9000000 0 0x10000<br>
> 0x82000000 0 0xc0000000 0 0xc0000000 0 0x20000000>;<br>
><br>
> Scanning PCI devices on bus 1<br>
> BusDevFun VendorId DeviceId Device Class Sub-Class<br>
> ______________________________<wbr>______________________________<wbr>_<br>
> 01.00.00 0x8086 0x1521 Network controller 0x00<br>
> 01.00.01 0x8086 0x1521 Network controller 0x00<br>
> Marvell>> pci bar 01.00.00<br>
> ID Base Size Width Type<br>
> ------------------------------<wbr>----------------------------<br>
> 0 0x000000c0000000 0x00000000100000 32 MEM<br>
><br>
> md.l ---> Abort<br>
><br>
> However I get a data abort.<br>
><br>
> Is this likely just a memory map setup problem in u-boot, or does this<br>
> larger BAR size require additional SMMU or other setup not done in u-boot?<br>
><br>
> The memory map document for Armada 8040 specify the PCIe range from<br>
> 0xf6000000 so my first guess is I need to modify the page table mapping?<br>
><br>
<br>
</div></div>Hello Frederik,<br>
<br>
There are several levels of physical address remapping in the 8040 SoC<br>
that all need to point to the PCIe controller. These settings are done<br>
in ARM Trusted Firmware IIRC, and I had to modify them to make a<br>
memory window of sufficient size (256 MB config space and 512 MB MMIO<br>
space) available for PCIe.<br>
<br></div></div>
______________________________<wbr>_________________<br>
Macchiato mailing list<br>
<a href="mailto:Macchiato@lists.einval.com" target="_blank">Macchiato@lists.einval.com</a><br>
<a href="https://lists.einval.com/cgi-bin/mailman/listinfo/macchiato" rel="noreferrer" target="_blank">https://lists.einval.com/cgi-b<wbr>in/mailman/listinfo/macchiato</a></blockquote></div><br></div>
</blockquote></div></div></div><br></div></div>
</blockquote></div><br></div></div>