1G RGMII on KR260

[russell-t]

Hello,

First off I just want to say thank you for making this resource available, it's a significant enabler for so many projects - thank you.

The issue I am having has to do with the eth_pcspma module. I am in the midst of creating a 1G RGMII example for the KR260, loosely following the discussion here where you suggested using the Nexys Video example as a starting point. In that example, an eth_mac_1g_rgmii_fifo module is instantiated in the fpga_core.v file.

However the VCU118 board uses the same PHY chip as the KR260, and in the VCU118/fpga_1g example, an eth_mac_1g_fifo is used in conjunction with a gig_ethernet_pcs_pma_0 module, which seems to translate RGMII into SGMII which is what the PHY chip expects (?). The pcs/pma module seems to require a license and is fairly annoying to deal with based on other issue threads I've looked at.

Is there a way around using this module, or is it necessary to hook into the DP83867 PHY chip?

[Alex Forencich]

Depends on how the PHY chip is wired up. If it's wired for RGMII, then you should be able to use the RGMII wrapper. If it's wired for SGMII, then at least as of right now you have to use the Xilinx PCS/PMA core. We might make an open-source 1000BASE-X module at some point, but there is no timeline on that at the moment.

For the KR260, I think the PL connected PHYs are all RGMII.

[russell-t]

Thanks for the quick reply. Yes, looking at the VCU118 schematic I now see that the PHY is wired in such a way that only supports SGMII, whereas the PHY on the KR260 is wired to support RGMII, and that the PHY itself supports both RGMII and SGMII configurations. Thanks for clarifying that.

In the VCU118 example,fpga.v contains a a state machine on lines 362 - 487 that seems to set the PHY up in SGMII mode, I am guessing this will need to be changed appropriately to set it up in RGMII. It is unclear to me what the purpose of the mdio_master module instantiation immediately following that section is doing. Could you elaborate a bit on this?

[Alex Forencich]

The state machine tells mdio_master what to do. And I suspect that you might not need to poke anything via MDIO for the PHY chip to work for RGMII, so try it without the init state machine first. IIRC the stuff that needs to be changed for the VCU118 design to work may be specific to SGMII.

[russell-t]

Understood, thank you. After taking a closer look at the data sheet for the PHY it does appear that by default the chip is set up for RGMII. Just to clarify, are you suggesting that mdio_master may be unneeded entirely? Based on your description it sounds like without the state machine there would be no data for the module to send.

[russell-t]

Documenting this for others who may see this. After creating a constraint for the 25MHz crystal oscillator and updating the MMCME2_BASE parameters accordingly, I am able to perform synthesis. However I am failing on implementation now with errors of SIM_DEVICE on cells phy_rx_ctl_idelay and phy_rxd_idelay_x not being specified and being set to ULTRASCALE_PLUS. It seems like maybe IDELAYE2 is not supported for Ultrascale+ and that I need to swap them out for IDELAYE3 modules. Does that seem right?

[Alex Forencich]

Yeah, you'll need to use the correct primitives for the target device. I guess that would be the IDELAYE3 for US+. I don't think I have an example that I can point you at for those, but presumably swapping those out should be pretty simple.

[russell-t]

After some debugging I've got GEM2 "working" after swapping to IDELAYE3's and also to an MMCME4_BASE. The only thing is that the E3 instances work a bit differently than the E2s, there is now the option to specify number of taps ("count" mode) or number of picoseconds ("time" mode) with the DELAY_VALUE. My understanding is that in "count" mode temperature/voltage compensation isn't used and an IDELAYCTRL isn't needed, so the delay is not super accurate. However in "count" mode I am getting UDP packets, but the majority of them are being dropped, but in "time" mode I get nothing at all. Trying to sort this out now.

[russell-t]

Got it working! Only issue now is that sometimes on the initial loading of the FPGA overlay, it doesn't work entirely or takes a few seconds to start working. I think this may be an issue with the start up sequence. The data sheet of the PHY specifies that RESET_N must be pulled low for at least 1 microsecond on startup, and in the reference design this is tied to the inverted LOCKED signal of the MMCME. I don't think there is any guarantee the MMCM takes at least 1 microsecond to initialize. Did you see any similar behavior with the VCU118?

[sahasam]

Hey, I've been following the discussion trying to replicate russell's accomplishment on my KR260. I've created a new vivado project, brought in the necessary files by referencing the VCU118 Example Design, NexysVideo Example Design, and vmayoral's KR260 SFP design. I've got it synthesizing and generating a bitstream after converting to MMCME4_BASE and IDELAYE3 and dragging in constraints. However, the LED's on the connector still aren't turning on. I'd like to check a few assumptions here, if you wouldn't mind. For now, I'm only concerned about GEM2 PL.

1. No Poking via MDIO is necessary. I checked the strap resistors in the schematic and I see the MDIO address is 0x02, Autonegotiation is enabled and set to advertise 10/100/1000, SGMII is disabled, and both TX/RX clock skew is set to 0. Is this configuration really enough that no connection via MDIO is necessary?

2. No connection to the PS is necessary. I'm using xmutil to load my designs on the KR260 after scp'ing the bitfile and dtbo. As a result, I need to create a hardware platform which requires some boilerplate in the hardware design. I want to do everything in PL because that's just my intended use case, but I'm starting to suspect Xilinx doesn't like that idea, especially on the Zynq platform. I would imagine it's possible, but do you know anything about this? EDIT: I realize I should explain a bit more here. I have fpga_core inside the fpga_eth rtl wrapper alongside clocking and reset logic. In fpga_core.v, I'm using the udp_complete module hoping I can do the netcat mirroring thing.
   

3. The IDELAYE3's for the RGMII rxd and rx_ctl signals don't require any delays. I have them because the discussion says I need them, and the example design for the NexysVideo has them (my rgmii reference). Here in the code the idelays just exist without really delaying anything. I carried that over to my design. This look okay to you?

IDELAYCTRL #(
    .SIM_DEVICE("ULTRASCALE")
)
idelayctrl_inst
(
    .REFCLK(clk_500mhz_int),
    .RST(rst_125mhz_int),       // async reset
    .RDY()
);


IDELAYE3 #(
    .SIM_DEVICE("ULTRASCALE_PLUS"),
    .DELAY_TYPE("FIXED"),
    .DELAY_FORMAT("COUNT"),
    .REFCLK_FREQUENCY(500.0)
)
phy_rxd_idelay_0
(
    .IDATAIN(phy_rxd[0]),
    .DATAOUT(phy_rxd_delay[0]),
    .DATAIN(1'b0),
    .CLK(1'b0),
    .CE(1'b0),
    .INC(1'b0),
    .LOAD(1'b0),
    .CNTVALUEIN(8'd0),
    .CNTVALUEOUT(),
    .CASC_RETURN(1'b0),
    .CASC_IN(1'b0),
    .CASC_OUT(),
    .RST(1'b0),
    .EN_VTC(1'b0)
);

Please tell me I'm wrong somewhere, then maybe I can make some progress.

PS For those who found this through google, here's my pin constraints for PL GEM2 RGMII on the kr260 that I found by cross referencing the schematic with the XDC

set_property -dict {LOC E1 IOSTANDARD LVCMOS18} [get_ports RGMII_td[0]]  ;   # som_240_1_d7   HPA01P
set_property -dict {LOC D1 IOSTANDARD LVCMOS18} [get_ports RGMII_td[1]]  ;   # som_240_1_d8   HPA01N
set_property -dict {LOC F2 IOSTANDARD LVCMOS18} [get_ports RGMII_td[2]]  ;   # som_240_1_d4   HPA02P
set_property -dict {LOC E2 IOSTANDARD LVCMOS18} [get_ports RGMII_td[3]]  ;   # som_240_1_d5   HPA02N
set_property -dict {LOC F1 IOSTANDARD LVCMOS18} [get_ports RGMII_tx_ctl] ;   # som_240_1_c4   HPA00_CCN
set_property -dict {LOC A2 IOSTANDARD LVCMOS18} [get_ports RGMII_txc]    ;   # som_240_1_a3   HPA06P_CLK
set_property -dict {LOC A1 IOSTANDARD LVCMOS18} [get_ports RGMII_rd[0]]  ;   # som_240_1_a4   HPA06N
set_property -dict {LOC B3 IOSTANDARD LVCMOS18} [get_ports RGMII_rd[1]]  ;   # som_240_1_b7   HPA07P
set_property -dict {LOC A3 IOSTANDARD LVCMOS18} [get_ports RGMII_rd[2]]  ;   # som_240_1_b8   HPA07N
set_property -dict {LOC B4 IOSTANDARD LVCMOS18} [get_ports RGMII_rd[3]]  ;   # som_240_1_c9   HPA08P
set_property -dict {LOC A4 IOSTANDARD LVCMOS18} [get_ports RGMII_rx_ctl] ;   # som_240_1_c10  HPA08N
set_property -dict {LOC D4 IOSTANDARD LVCMOS18} [get_ports RGMII_rxc]    ;   # som_240_1_d10  HPA09P_CLK

[russell-t]

Hi @sahasam,

Edit: I realized my reply doesn't completely correlate with the questions you asked. Rearranging my answers to match your questions by numbers.

1. The MDIO seems to be necessary. Without it the PHY seems to be in a weird state after loading the FPGA overlay where only some of the received packets get properly looped back. I have the state machine sending a software reset command (I believe it's 0x8000 to address 0x1F)
2. This is fine. You just need the Zynq MPSOC. You can even disable all inputs and outputs on it so it isn't connected to anything else.
3. I found the IDELAYE3's to be necessary on the phy_rxd and phy_rx_ctl pins. To get them working, this is the configuration I found that works. Note the delay format is TIME instead of COUNT, and the delay value is 240. EN_VTC must also be pulled high, with the other pins shown below pulled low.

.CASCADE("NONE"),
.REFCLK_FREQUENCY(300.0),
.DELAY_FORMAT("TIME"),
.IS_RST_INVERTED(1'b0),
.DELAY_TYPE("FIXED"),
.SIM_DEVICE("ULTRASCALE_PLUS"),
.DELAY_VALUE(240)

.CE(1'b0),
.DATAIN(1'b0),
.EN_VTC(1'b1),
.INC(1'b0),
.LOAD(1'b0),
.RST(1'b0)

Where .IDATAIN is connected to the input signal and .DATAOUT is the output signal. All other inputs/outputs not listed were left unconnected.

4. One more thing - you need to add the reset_n connection to the PHY (LOC is B1) and ensure this pin is driven HIGH (I have it set as the inverted signal of the reset created by the MMCME4/sync_resets. Only after adding this was I able to see lights on the RJ45.

Hope this helps.