How to add a new board of FPGA device (alinx_ax7010)

Question

How to add a new board of FPGA device (alinx_ax7010)

ggangliu opened this issue 2 years ago · comments

I am trying to add a new FPGA device. Here is the question I encountered and as a handbook as well.
out.log

gang@ubuntu:~/CFU-Playground/proj/proj_zynq7010$ make bitstream USE_VIVADO=1 IGNORE_TIMING=1 TARGET=xilinx_zynq_xc7z010 > out.log 
INFO:SoC:        __   _ __      _  __  
INFO:SoC:       / /  (_) /____ | |/_/  
INFO:SoC:      / /__/ / __/ -_)>  <    
INFO:SoC:     /____/_/\__/\__/_/|_|  
INFO:SoC:  Build your hardware, easily!
INFO:SoC:--------------------------------------------------------------------------------
INFO:SoC:Creating SoC... (2022-02-28 09:28:22)
INFO:SoC:--------------------------------------------------------------------------------
INFO:SoC:FPGA device : xc7z010clg400-1.
INFO:SoC:System clock: 100.000MHz.
INFO:SoCBusHandler:Creating Bus Handler...
INFO:SoCBusHandler:32-bit wishbone Bus, 4.0GiB Address Space.
INFO:SoCBusHandler:Adding reserved Bus Regions...
INFO:SoCBusHandler:Bus Handler created.
INFO:SoCCSRHandler:Creating CSR Handler...
INFO:SoCCSRHandler:32-bit CSR Bus, 32-bit Aligned, 16.0KiB Address Space, 2048B Paging, big Ordering (Up to 32 Locations).
INFO:SoCCSRHandler:Adding reserved CSRs...
INFO:SoCCSRHandler:CSR Handler created.
[out.log](https://github.com/google/CFU-Playground/files/8155475/out.log)

INFO:SoCIRQHandler:Creating IRQ Handler...
INFO:SoCIRQHandler:IRQ Handler (up to 32 Locations).
INFO:SoCIRQHandler:Adding reserved IRQs...
INFO:SoCIRQHandler:IRQ Handler created.
INFO:SoC:--------------------------------------------------------------------------------
INFO:SoC:Initial SoC:
INFO:SoC:--------------------------------------------------------------------------------
INFO:SoC:32-bit wishbone Bus, 4.0GiB Address Space.
INFO:SoC:32-bit CSR Bus, 32-bit Aligned, 16.0KiB Address Space, 2048B Paging, big Ordering (Up to 32 Locations).
INFO:SoC:IRQ Handler (up to 32 Locations).
INFO:SoC:--------------------------------------------------------------------------------
INFO:SoCBusHandler:io0 Region added at Origin: 0x80000000, Size: 0x80000000, Mode: RW, Cached: False Linker: False.
INFO:SoC:CPU overriding rom mapping from 0x0 to 0x0.
INFO:SoC:CPU overriding sram mapping from 0x1000000 to 0x10000000.
INFO:SoC:CPU overriding main_ram mapping from 0x40000000 to 0x40000000.
INFO:SoCBusHandler:cpu_bus0 added as Bus Master.
INFO:SoCBusHandler:cpu_bus1 added as Bus Master.
INFO:SoCBusHandler:rom Region added at Origin: 0x00000000, Size: 0x00020000, Mode: R, Cached: True Linker: False.
INFO:SoCBusHandler:rom added as Bus Slave.
INFO:SoC:RAM rom added Origin: 0x00000000, Size: 0x00020000, Mode: R, Cached: True Linker: False.
INFO:SoCBusHandler:sram Region added at Origin: 0x10000000, Size: 0x00002000, Mode: RW, Cached: True Linker: False.
INFO:SoCBusHandler:sram added as Bus Slave.
INFO:SoC:RAM sram added Origin: 0x10000000, Size: 0x00002000, Mode: RW, Cached: True Linker: False.
INFO:SoCIRQHandler:uart IRQ allocated at Location 0.
INFO:SoCIRQHandler:timer0 IRQ allocated at Location 1.
INFO:S7PLL:Creating S7PLL, speedgrade -1.
INFO:S7PLL:Registering Single Ended ClkIn of 100.00MHz.
INFO:S7PLL:Creating ClkOut0 sys of 100.00MHz (+-10000.00ppm).
INFO:S7PLL:Config:
divclk_divide : 1
clkout0_freq  : 100.00MHz
clkout0_divide: 16
clkout0_phase : 0.00°
vco           : 1600.00MHz
clkfbout_mult : 16
INFO:SoCBusHandler:csr Region added at Origin: 0xf0000000, Size: 0x00010000, Mode: RW, Cached: False Linker: False.
INFO:SoCBusHandler:csr added as Bus Slave.
INFO:SoCCSRHandler:bridge added as CSR Master.
INFO:SoCBusHandler:Interconnect: InterconnectShared (2 <-> 3).
INFO:SoCCSRHandler:ctrl CSR allocated at Location 0.
INFO:SoCCSRHandler:identifier_mem CSR allocated at Location 1.
INFO:SoCCSRHandler:leds CSR allocated at Location 2.
INFO:SoCCSRHandler:timer0 CSR allocated at Location 3.
INFO:SoCCSRHandler:uart CSR allocated at Location 4.
INFO:SoC:--------------------------------------------------------------------------------
INFO:SoC:Finalized SoC:
INFO:SoC:--------------------------------------------------------------------------------
INFO:SoC:32-bit wishbone Bus, 4.0GiB Address Space.
IO Regions: (1)
io0                 : Origin: 0x80000000, Size: 0x80000000, Mode: RW, Cached: False Linker: False
Bus Regions: (3)
rom                 : Origin: 0x00000000, Size: 0x00020000, Mode: R, Cached: True Linker: False
sram                : Origin: 0x10000000, Size: 0x00002000, Mode: RW, Cached: True Linker: False
csr                 : Origin: 0xf0000000, Size: 0x00010000, Mode: RW, Cached: False Linker: False
Bus Masters: (2)
- cpu_bus0
- cpu_bus1
Bus Slaves: (3)
- rom
- sram
- csr
INFO:SoC:32-bit CSR Bus, 32-bit Aligned, 16.0KiB Address Space, 2048B Paging, big Ordering (Up to 32 Locations).
CSR Locations: (5)
- ctrl           : 0
- identifier_mem : 1
- leds           : 2
- timer0         : 3
- uart           : 4
INFO:SoC:IRQ Handler (up to 32 Locations).
IRQ Locations: (2)
- uart   : 0
- timer0 : 1
INFO:SoC:--------------------------------------------------------------------------------
INFO:SoC:Initializing ROM rom with contents (Size: 0x5108).
INFO:SoC:Auto-Resizing ROM rom from 0x20000 to 0x5108.



ERROR: [DRC UCIO-1] Unconstrained Logical Port: 2 out of 7 logical ports have no user assigned specific location constraint (LOC). This may cause I/O contention or incompatibility with the board power or connectivity affecting performance, signal integrity or in extreme cases cause damage to the device or the components to which it is connected. To correct this violation, specify all pin locations. This design will fail to generate a bitstream unless all logical ports have a user specified site LOC constraint defined.  To allow bitstream creation with unspecified pin locations (not recommended), use this command: set_property SEVERITY {Warning} [get_drc_checks UCIO-1].  NOTE: When using the Vivado Runs infrastructure (e.g. launch_runs Tcl command), add this command to a .tcl file and add that file as a pre-hook for write_bitstream step for the implementation run.  Problem ports: serial_rx, and serial_tx.
ERROR: [Vivado 12-1345] Error(s) found during DRC. Bitgen not run.
Traceback (most recent call last):
  File "./common_soc.py", line 57, in <module>
    main()
  File "./common_soc.py", line 53, in main
    workflow.run()
  File "/home/ggang/CFU-Playground/soc/board_specific_workflows/general.py", line 121, in run
    soc_builder = self.build_soc(soc)
  File "/home/ggang/CFU-Playground/soc/board_specific_workflows/general.py", line 98, in build_soc
    soc_builder.build(run=self.args.build, **kwargs)
  File "/home/ggang/CFU-Playground/third_party/python/litex/litex/soc/integration/builder.py", line 330, in build
    vns = self.soc.build(build_dir=self.gateware_dir, **kwargs)
  File "/home/ggang/CFU-Playground/third_party/python/litex/litex/soc/integration/soc.py", line 1139, in build
    return self.platform.build(self, *args, **kwargs)
  File "/home/ggang/CFU-Playground/third_party/python/litex/litex/build/xilinx/platform.py", line 55, in build
    return self.toolchain.build(self, *args, **kwargs)
  File "/home/ggang/CFU-Playground/third_party/python/litex/litex/build/xilinx/vivado.py", line 372, in build
    _run_script(script)
  File "/home/ggang/CFU-Playground/third_party/python/litex/litex/build/xilinx/vivado.py", line 101, in _run_script
    raise OSError("Error occured during Vivado's script execution.")
OSError: Error occured during Vivado's script execution.
make[1]: *** [/home/ggang/CFU-Playground/soc/common_soc.mk:115: build/xilinx_zynq_xc7z010.proj_zynq7010/gateware/xilinx_zynq_xc7z010.bit] Error 1
make: *** [../proj.mk:287: bitstream] Error 2

TCal · Answer 1 · Tue Mar 01 2022 02:00:43 GMT+0800 (China Standard Time)

Hi @ggangliu , thanks for trying this! I recently got a Digilent Arty Z7 board with a z7010 myself.

Did you add the board platform/target files for "xilinx_zynq_xc7z010" yourself? Is that an eval board from Xilinx? I don't see it in https://github.com/litex-hub/litex-boards/tree/master/litex_boards/platforms .

Yonggang Liu · Answer 2 · Tue Mar 01 2022 10:09:53 GMT+0800 (China Standard Time)

Hi @ggangliu , thanks for trying this! I recently got a Digilent Arty Z7 board with a z7010 myself.

Did you add the board platform/target files for "xilinx_zynq_xc7z010" yourself? Is that an eval board from Xilinx? I don't see it in https://github.com/litex-hub/litex-boards/tree/master/litex_boards/platforms .

Thanks for reply. Yes I have added the board platform/target files for it in local Repo. It is from Xilinx, I saw the device ID is the same with digilent_zybo_z7.py. So even I can program "digilent_zybo_z7.bit" file into my board. So I almost copy and modify my board based on digilent_zybo_z7 board. I will submit after I build it successful at least. I think I am really close to success. ^_^

Is there anywhere need to be changed also unless litex-boards?

TCal · Answer 3 · Tue Mar 01 2022 16:30:24 GMT+0800 (China Standard Time)

@ggangliu does the basic LiteX build work with your board files? Either using the "PS" ARM cores, or using a soft CPU built on the FPGA fabric?

TCal · Answer 4 · Wed Mar 02 2022 06:59:38 GMT+0800 (China Standard Time)

@ggangliu , the issue you have with the tty connection (serial_tx, serial_rx) is probably similar to what I see on my Arty Z7. On my Arty, the USB cable connects to an FTDI chip that then connects to the PS, not to the PL (FPGA).

On other boards, with plain FPGAs, there usually dedicated FPGA I/Os for serial_rx / serial_tx that connect to the FTDI chip and then USB. That's what LiteX is expecting, but in our case, those pins aren't declared in the FPGA platform file.

One solution is to set up your own UART/USB conversion by buying a small board like this https://www.amazon.com/gp/product/B00IJXZQ7C/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&psc=1 (there are MANY alternatives on Amazon / Alibaba / electronics supply places). You will then need to pick the FPGA IOs to use, add them to your LiteX platform file, and then set up the serial port in the target file. And of course wire the converter board to your Zynq board (just 3 wires are needed). Zybo does something similar using its PMOD connector: https://github.com/litex-hub/litex-boards/blob/master/litex_boards/platforms/digilent_zybo_z7.py#L69-L77

Yonggang Liu · Answer 5 · Wed Mar 02 2022 12:06:40 GMT+0800 (China Standard Time)

@tcal-x Really thanks. I already have a UART/USB conversion, I can try it soon. It is really glad to know this. Thanks again.

By the way, why here has "serial" and "usb_uart" two definitions? I think only one is needed, right?

Yonggang Liu · Answer 6 · Thu Mar 03 2022 01:40:26 GMT+0800 (China Standard Time)

@ggangliu , the issue you have with the tty connection (serial_tx, serial_rx) is probably similar to what I see on my Arty Z7. On my Arty, the USB cable connects to an FTDI chip that then connects to the PS, not to the PL (FPGA).

On other boards, with plain FPGAs, there usually dedicated FPGA I/Os for serial_rx / serial_tx that connect to the FTDI chip and then USB. That's what LiteX is expecting, but in our case, those pins aren't declared in the FPGA platform file.

One solution is to set up your own UART/USB conversion by buying a small board like this https://www.amazon.com/gp/product/B00IJXZQ7C/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&psc=1 (there are MANY alternatives on Amazon / Alibaba / electronics supply places). You will then need to pick the FPGA IOs to use, add them to your LiteX platform file, and then set up the serial port in the target file. And of course wire the converter board to your Zynq board (just 3 wires are needed). Zybo does something similar using its PMOD connector: https://github.com/litex-hub/litex-boards/blob/master/litex_boards/platforms/digilent_zybo_z7.py#L69-L77

I have tested, it is really like what you said. I use FPGA pins as Serial ports, building is successfully. And LEDs is flashing.

TCal · Answer 7 · Thu Mar 03 2022 06:44:49 GMT+0800 (China Standard Time)

I have tested, it is really like what you said. I use FPGA pins as Serial ports, building is successfully. And LEDs is flashing.

That's great! Have you been able to connect over your new USB/serial to interact with the LiteX BIOS?

TCal · Answer 8 · Thu Mar 03 2022 06:48:00 GMT+0800 (China Standard Time)

I have tested, it is really like what you said. I use FPGA pins as Serial ports, building is successfully. And LEDs is flashing.

That's great! Have you been able to connect over your new USB/serial to interact with the LiteX BIOS?

P.S. @ggangliu I have NOT tried this yet -- I've only gotten as far as you have -- getting the LED Chaser blinking. See litex-hub/litex-boards#358 .

Yonggang Liu · Answer 9 · Thu Mar 03 2022 10:54:19 GMT+0800 (China Standard Time)

I have tested, it is really like what you said. I use FPGA pins as Serial ports, building is successfully. And LEDs is flashing.

That's great! Have you been able to connect over your new USB/serial to interact with the LiteX BIOS?

P.S. @ggangliu I have NOT tried this yet -- I've only gotten as far as you have -- getting the LED Chaser blinking. See litex-hub/litex-boards#358 .

Yes, I have got the right output of BIOS with baud rate 57600. The output looks very beautiful.

Yonggang Liu · Answer 10 · Fri Mar 04 2022 00:16:32 GMT+0800 (China Standard Time)

A new question:
When programming bit file, JTAG only try to program the first device, but the first device is the ARM core (PS), so cause programming failed. I have checked the related source, I found that the default value of device is 0, I guess this is the reason why JTAG only try the first device. Then I changed it to 1, I can program PL successfully. And UART also has the correct output of BIOS.

I have raised an issue:
enjoy-digital/litex#1231

TCal · Answer 11 · Fri Mar 04 2022 01:04:17 GMT+0800 (China Standard Time)

Hi @ggangliu , those are great findings. I never would have thought to try 1/2 speed baud rate! I hope I can find time to see what the status is with my build on Arty Z7.

To clarify the JTAG discussion --- I thought you were able to program PL before (to get the blinking LED). Are you saying that you need to make a change to program the PL correctly?

Ah, the device value should probably be overridden in the board target file, not in the central xilinx/programmer.py file. Here is how it works with the target file for Arty Z7:

prog.load_bitstream(os.path.join(builder.gateware_dir, soc.build_name + ".bit"), device=1)

https://github.com/litex-hub/litex-boards/blob/master/litex_boards/targets/digilent_arty_z7.py#L131

Yonggang Liu · Answer 12 · Fri Mar 04 2022 10:14:42 GMT+0800 (China Standard Time)

Hi @ggangliu , those are great findings. I never would have thought to try 1/2 speed baud rate! I hope I can find time to see what the status is with my build on Arty Z7.

To clarify the JTAG discussion --- I thought you were able to program PL before (to get the blinking LED). Are you saying that you need to make a change to program the PL correctly?

Ah, the device value should probably be overridden in the board target file, not in the central xilinx/programmer.py file. Here is how it works with the target file for Arty Z7:
prog.load_bitstream(os.path.join(builder.gateware_dir, soc.build_name + ".bit"), device=1)
https://github.com/litex-hub/litex-boards/blob/master/litex_boards/targets/digilent_arty_z7.py#L131

Really thanks @tcal-x. The first time success, I was using Vivado GUI to program, so I can select specific device. Yesterday I tried to use command line "make prog". Appreciate again.

Yonggang Liu · Answer 13 · Sat Mar 12 2022 19:37:31 GMT+0800 (China Standard Time)

Hi @tcal-x,
A new question, I was running this command "make load TTY=/dev/ttyUSB0 TARGET=xilinx_zynq_xc7z010", it encountered below issue. Could you please give me any suggestion? Thanks.

  CXX schema_utils.cc	schema_utils.o
  CXX tflite.cc	tflite.o
  CXX tflite_unit_tests.cc	tflite_unit_tests.o
  AS  crt0-vexriscv.S	crt0-vexriscv.o
  LD       software.elf
/home/ggang/Downloads/riscv64-unknown-elf-gcc-10.1.0-2020.08.2-x86_64-linux-ubuntu14/bin/../lib/gcc/riscv64-unknown-elf/10.1.0/../../../../riscv64-unknown-elf/bin/ld:/home/ggang/CFU-Playground/proj/proj_zynq7010/build/ld/linker.ld:20: warning: memory region `main_ram' not declared
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:163: software.elf] Error 1
make[1]: Leaving directory '/home/ggang/CFU-Playground/proj/proj_zynq7010/build'
make: *** [../proj.mk:214: /home/ggang/CFU-Playground/proj/proj_zynq7010/build/software.bin] Error 2

What I have done

I have checked vexriscv core.py file, it looks no proplem

./third_party/python/litex/litex/soc/cores/cpu/vexriscv/core.py     def mem_map(self):
        return {
            "rom":            0x00000000,
            "sram":           0x10000000,
            "main_ram":       0x40000000,
            "csr":            0xf0000000,
            "vexriscv_debug": 0xf00f0000,
        }

regions.ld shows like below, it seems be generated automatically. And no "main_ram" section definition.

MEMORY {
        rom : ORIGIN = 0x00000000, LENGTH = 0x00020000
        sram : ORIGIN = 0x10000000, LENGTH = 0x00002000
        csr : ORIGIN = 0xf0000000, LENGTH = 0x00010000
}

linker.ld in build/ld folder like below, all sections are using "main_ram" as target area.

INCLUDE output_format.ld
ENTRY(_start)

__DYNAMIC = 0;

INCLUDE regions.ld


SECTIONS
{
        .text :
        {
                _ftext = .;
                *(.text.start)
                *(.text .stub .text.* .gnu.linkonce.t.*)
                _etext = .;
        } > main_ram

TCal · Answer 14 · Tue Mar 15 2022 00:19:50 GMT+0800 (China Standard Time)

Hi @ggangliu , yes, that is a real issue. I see the same thing with my Arty Z7 board -- no main_ram region.

But we can fix that by adding a new RAM! We could either split the existing rom region -- but then I think we'd need to change the origin of the main_ram region to be 0x10000 (with the original rom using 0x00000 - 0x10000). There's a precedent with what they have done with Fomu, which now splits the psram into two regions: https://github.com/litex-hub/litex-boards/blob/master/litex_boards/targets/kosagi_fomu.py#L93-L106

But, we could try adding a new region for main_ram made out of unused BRAMs. You can add a RAM on the Wishbone bus using this:

self.add_ram("main_ram", 0x40000000, 0x10000)

I haven't tried this yet, but hopefully later today.

TCal · Answer 15 · Tue Mar 15 2022 05:24:38 GMT+0800 (China Standard Time)

@ggangliu , I was able to add the new memory region, but I couldn't really test it since I couldn't get the tty / uart to work with the PMOD adapter.

Yonggang Liu · Answer 16 · Tue Mar 15 2022 15:10:01 GMT+0800 (China Standard Time)

@ggangliu , I was able to add the new memory region, but I couldn't really test it since I couldn't get the tty / uart to work with the PMOD adapter.

putty and litex_term are able to connect via PMOD adapter with sudo command, but sometime it works on /dev/ttyUSB0, sometime it is /dev/ttyUSB1, it is not stable. And default baud rate is 1843200. Some UARTs tool don't support this baud rate.

Yonggang Liu · Answer 17 · Tue Mar 15 2022 15:28:42 GMT+0800 (China Standard Time)

Hi @ggangliu , yes, that is a real issue. I see the same thing with my Arty Z7 board -- no main_ram region.

But we can fix that by adding a new RAM! We could either split the existing rom region -- but then I think we'd need to change the origin of the main_ram region to be 0x10000 (with the original rom using 0x00000 - 0x10000). There's a precedent with what they have done with Fomu, which now splits the psram into two regions: https://github.com/litex-hub/litex-boards/blob/master/litex_boards/targets/kosagi_fomu.py#L93-L106

But, we could try adding a new region for main_ram made out of unused BRAMs. You can add a RAM on the Wishbone bus using this:
self.add_ram("main_ram", 0x40000000, 0x10000)
I haven't tried this yet, but hopefully later today.

Thanks tcal-x, this method is able to add any custom section, it is very flexible.

Through analyzing source code, I found there is a simpler way to enable it that we only need to add parameter(integrated_main_ram_size = 0x4000) in board target file, like below code.

        # SoCCore ----------------------------------------------------------------------------------
        SoCCore.__init__(self, platform, sys_clk_freq,
            ident          = "LiteX SoC on ECP5 Evaluation Board",
            integrated_main_ram_size = 0x4000,
            **kwargs)

And also found a reference here. https://github.com/litex-hub/litex-boards/blob/08a79fa3ac68d1ddff0ba58432e86351efbbf100/litex_boards/targets/lattice_ecp5_vip.py#L104-L105

Yonggang Liu · Answer 18 · Tue Mar 15 2022 20:24:47 GMT+0800 (China Standard Time)

Hi @tcal-x , How can I execute "make load" with demo.bin? Right now I don't how to build demo.bin. I have to start with demo program because my board has no sufficient memory to store tensorflow image.

ggang@ubuntu:~/cfu-playground/proj/proj_zynq7010_cfu$ litex_bare_metal_demo --build-path=/build/arty/
litex_bare_metal_demo: command not found