C5 SoC Computer System

Bus Lecture Notes Updated

We appended exercises on UART and CLINT to the bus section of our lecture notes on November 29, 2023. Completing these exercises will be beneficial for the subsequent SoC integration.

After implementing the bus, we can connect the NPC to the OSOC SoC environment, preparing for tape-out! SoC stands for System On Chip, which means that an SoC contains not just a processor, but also numerous peripheral devices, as well as the bus that connects the processor to these peripherals. Here, we regard memory as a type of device in a broad sense, since, for an SoC, memory and other narrowly-defined devices are indistinguishable — they are all addressable ranges of address space.

ysyxSoC

We provide an SoC environment that can run on Verilator, called ysyxSoC. We allow you to integrate with ysyxSoC early for two reasons: on the one hand, to help everyone learn about its details, and on the other hand, to test your NPC in the SoC environment as early as possible, thereby shortening the time between completing the tape-out assessment and submitting your code. Of course, after integrating with ysyxSoC, you will still need to complete some optimization work to meet the B Stage tape-out requirements.

ysyxSoC Introduction

First, we present the peripheral devices included in ysyxSoC and their corresponding address spaces.

Device	Address Space
CLINT	0x0200_0000~0x0200_ffff
SRAM	0x0f00_0000~0x0fff_ffff
UART16550	0x1000_0000~0x1000_0fff
SPI master	0x1000_1000~0x1000_1fff
GPIO	0x1000_2000~0x1000_200f
PS2	0x1001_1000~0x1001_1007
MROM	0x2000_0000~0x2000_0fff
VGA	0x2100_0000~0x211f_ffff
Flash	0x3000_0000~0x3fff_ffff
ChipLink MMIO	0x4000_0000~0x7fff_ffff
PSRAM	0x8000_0000~0x9fff_ffff
SDRAM	0xa000_0000~0xbfff_ffff
ChipLink MEM	0xc000_0000~0xffff_ffff
Reserved	Others

In addition to AXI, the figure also involves buses such as APB, wishbone and SPI. However, these buses are simpler than AXI, even simpler than AXI4-Lite. Since you already know AXI4-Lite, learning these bus protocols will not be difficult, and you can consult the relevant manuals whenever necessary.

Some devices and address spaces may change in the future

In order to achieve a better display effect, the OSOC project team is redesigning the SoC. Some devices and address spaces may change in the future, and the final device address space allocation is subject to the tape-out version. However, this does not affect your current learning, and you can safely ignore this situation.

Get the source code of ysyxSoC

You need to clone the ysyxSoC project:

cd ysyx-workbench
git clone git@github.com:OSCPU/ysyxSoC.git

Next, you will use the devices provided by ysyxSoC for simulation, to verify that the NPC can correctly access the devices in the SoC. We will introduce how to integrate it below.

It should be noted that there are still some differences between ysyxSoC and the SoC used in the final tape-out. Therefore, passing the ysyxSoC tests does not mean that it will eventually pass the tests of the tape-out SoC simulation environment. Even so, the ysyxSoC project can still help expose some problems in advance. If problems still arise when integrating with the tape-out SoC later, you can focus on the impact caused by the differences between the two.

For everyone, there are two parts of the ysyxSoC project that deserve attention. The first part is the bus of ysyxSoC, which we mainly implement using the diplomacy framework of the open source community rocket-chip project, with the relevant code located in the ysyxSoC/src/ directory. With diplomacy, we can easily connect a device with a bus interface to ysyxSoC. For example, we only need to change the following two lines of Chisel code to instantiate an AXI-interface MROM device, specify its address space as 0x2000_0000~0x2000_0fff, and connect it to the downstream of the AXI Xbar. If we were to use the traditional Verilog approach, merely declaring the ports would add nearly 100 lines of code, not to mention the modifications to the AXI Xbar.

diff --git a/src/SoC.scala b/src/SoC.scala
index dd84776c..758fb8d1 100644
--- a/src/SoC.scala
+++ b/src/SoC.scala
@@ -39,9 +39,10 @@ class ysyxSoCASIC(implicit p: Parameters) extends LazyModule {
     AddressSet.misaligned(0x10001000, 0x1000) ++    // SPI controller
     AddressSet.misaligned(0x30000000, 0x10000000)   // XIP flash
   ))
+  val lmrom = LazyModule(new AXI4MROM(AddressSet.misaligned(0x20000000, 0x1000)))

   List(lspi.node, luart.node).map(_ := apbxbar)
-  List(chiplinkNode, apbxbar := AXI4ToAPB()).map(_ := xbar)
+  List(chiplinkNode, apbxbar := AXI4ToAPB(), lmrom.node).map(_ := xbar)
   xbar := cpu.masterNode

   override lazy val module = new Impl

The second part is the devices of ysyxSoC. We have collected some open source projects of device controllers, with the relevant code located in the ysyxSoC/perip/ directory. Some devices are implemented by directly instantiating IP from the rocket-chip project; these devices are not located in the ysyxSoC/perip/ directory, and you can refer to the relevant code in ysyxSoC/src/ for details.

Integration into ysyxSoC

Since the SoC contains multiple devices, the properties of these devices may differ, which will introduce some new problems. For example, ysyxSoC/perip/uart16550/rtl/uart_defines.v contains the following code:

// Register addresses
`define UART_REG_RB `UART_ADDR_WIDTH'd0  // receiver buffer
`define UART_REG_IE `UART_ADDR_WIDTH'd1  // Interrupt enable
`define UART_REG_II `UART_ADDR_WIDTH'd2  // Interrupt identification
`define UART_REG_LC `UART_ADDR_WIDTH'd3  // Line Control

The above code defines the addresses of some device registers in the UART. Since the UART is located at 0x1000_0000, the addresses of the four registers above are 0x1000_0000, 0x1000_0001, 0x1000_0002, and 0x1000_0003. Assuming that the UART is connected to the Xbar through the AXI4-Lite bus, consider reading the contents of the receiver buffer through AXI4-Lite. Obviously, the araddr signal should be 0x1000_0000, but if you want to read 4 bytes, will the contents of the three device registers behind it also be read out at the same time?

We previously did not consider the question of "how many bytes to read", because reading the memory does not change the state of the data stored in it. Therefore, no matter how many bytes the CPU expects to read, the bus can read 4 or 8 bytes at a time and let the CPU select the target data from them. This even helps some CPUs with caches improve performance: reading data from memory once generally takes a long time, so if the bus bandwidth can be fully utilized to read more data at once, the number of future actual memory accesses may be reduced.

However, for device accesses, the above premise no longer holds: accessing a device register may change the state of the device! This means that, for a device, reading 1 byte and reading 4 bytes may ultimately lead to different behaviors. If we do not access the device registers according to their conventions, the device may enter an unpredictable state. Therefore, when accessing devices through the bus, we need to handle this issue carefully.

However, the AXI4-Lite bus cannot solve the above problem: its AR channel does not have enough signals to encode the read length information, so the device can only assume that the actual data bit width is the same as the AXI4-Lite bus data bit width. Therefore, if a single read request on the AXI4-Lite bus covers multiple device registers, it may cause errors in the device state. For this very reason, not all devices are suitable for connection through the AXI4-Lite bus.

For example, the above UART cannot be connected through an AXI4-Lite bus with a data bit width of 32 bits, because the interval between the device registers in the UART is only 1 byte, which means that reading one device register through AXI4-Lite will also affect the states of the corresponding device registers, which is not what we expect. For another UART whose device register address space is as follows, it can be connected through an AXI4-Lite bus with a data bit width of 32 bits, because the interval between these registers is 4 bytes, which is just enough to read one of the registers without affecting the states of the adjacent registers.

// Register addresses
`define UART_REG_RB `UART_ADDR_WIDTH'd0  // receiver buffer
`define UART_REG_IE `UART_ADDR_WIDTH'd4  // Interrupt enable
`define UART_REG_II `UART_ADDR_WIDTH'd8  // Interrupt identification
`define UART_REG_LC `UART_ADDR_WIDTH'd12 // Line Control

In order to solve the above problems of AXI4-Lite, the full AXI bus protocol uses the arsize/awsize signals to indicate the actual data bit width, and introduces the concept of "narrow transfer" to describe the situation where "the actual data bit width is smaller than the bus data bit width". These two concepts of "data bit width" are not entirely identical. Specifically, the bus data bit width is statically determined during hardware design; it represents the maximum data bit width of one bus transfer and is also used to compute the theoretical bandwidth of the bus. The actual data bit width (i.e., the value of the arsize/awsize signals) is dynamically determined by the bit width information in the software's memory access instructions, and it represents the actual data bit width of one bus transfer. For example, the lb instruction accesses only 1 byte, while the lw instruction accesses 4 bytes.

With the arsize/awsize signals, the device can learn the actual data bit width that the software needs to access, so that even when the addresses of several device registers are densely packed, it can access only one of them, avoiding accidental changes to the device state.

Generate the Verilog code for ysyxSoC

First, you need to perform some configuration and initialization:

1. Install mill according to the mill documentation
   • You can verify the installation with mill --version. In addition, the rocket-chip project requires mill to be version 0.11 or higher. If you find that your mill version does not meet the requirement, please install the latest version of mill
2. Run make dev-init under the ysyxSoC/ directory to fetch the rocket-chip project

The above two steps only need to be done once. After the configuration is complete, run make verilog under the ysyxSoC/ directory, and the generated Verilog file will be located at ysyxSoC/build/ysyxSoCFull.v.

Integration into ysyxSoC

Integrate the NPC into ysyxSoC by following the steps below in order:

1. According to the master bus in ysyxSoC/spec/cpu-interface.md, extend the previously implemented AXI4-Lite protocol to the full AXI4 protocol

2. Adjust the NPC top-level interface so that it is completely consistent with the interface naming convention in ysyxSoC/spec/cpu-interface.md, including signal direction, naming, and data bit width

   • For unused top-level output ports, assign them the constant 0
   • For unused top-level input ports, leave them floating

3. Add all .v files under the ysyxSoC/perip directory and its subdirectories to verilator's Verilog file list

4. Add the two directories ysyxSoC/perip/uart16550/rtl and ysyxSoC/perip/spi/rtl to verilator's include search paths

   • For how to add them, please RTFM (man verilator or verilator's official manual)
     • If you have never looked up verilator's options, we suggest that you take this opportunity to carefully read the argument summary in the manual — you may well discover some new treasures

5. Add --timescale "1ns/1ns" and --no-timing to the verilator compilation options

6. Add ysyxSoC/build/ysyxSoCFull.v to verilator's Verilog file list

7. Set the ysyxSoCFull module (defined in ysyxSoC/build/ysyxSoCFull.v) as the top-level module for verilator simulation

8. Change the ysyx_00000000 module name in ysyxSoC/build/ysyxSoCFull.v to the module name of your processor

   • Note that your processor module should not contain the SRAM and UART with the AXI4-Lite interface from the earlier exercises; we will use the memory and UART in ysyxSoC to replace them
   • However, your processor module should contain CLINT, since it will be used as a module in the tape-out project, and ysyxSoC does not include it

9. Add the following content to the simulation cpp file, to solve the problem of flash_read and mrom_read not being found during linking

   extern "C" void flash_read(int32_t addr, int32_t *data) { assert(0); }
   extern "C" void mrom_read(int32_t addr, int32_t *data) { assert(0); }
   

10. Add the statement Verilated::commandArgs(argc, argv); in the main function of the simulation environment, at the position before the simulation starts, to solve the runtime error reported by the plusargs functionality

11. Compile the simulation executable through verilator

    • If you encounter a combinational loop error, please modify your RTL code yourself

12. Try to start the simulation. You will observe that the code enters the main loop of the simulation, but the NPC has no valid output. We will solve this problem next

There are also some step instructions related to code inspection in ysyxSoC, but since you will still improve the NPC later, we will ask you to conduct the code inspection before the assessment. If you are interested, you may also carry out the code inspection now; we do not make it mandatory.

Next, we will introduce the devices in ysyxSoC one by one, and how to let programs use them. Some tasks will require you to implement or enhance some device modules at the RTL level, and for most of these tasks, you can choose to complete them with Chisel or Verilog. In particular, if you choose Chisel, you will still need to read some Verilog code to help you complete the tasks.

The simplest SoC

Recall two of the elements of TRM: there is a program that can be executed, and output is possible. In the previous simulation process, both of these were realized through the simulation environment: the simulation environment placed the program's image file into memory, so by the time the NPC fetches the first instruction, the program is already in memory; for output, we used the DPI-C function pmem_read() to call functions of the simulation environment, and achieved output through the putchar() function of the simulation environment. However, in a real SoC, after the board is powered on there is no simulation environment or runtime environment to provide the above functions, so these basic functions need to be implemented in hardware.

Program storage

First, we need to consider where to put the program. General-purpose memories are volatile, such as SRAM and DRAM, which do not contain valid data when powered on. If the CPU directly fetches and executes instructions from memory after power-on, what the memory reads out is undefined, so the behavior of the entire system is also undefined, making it impossible for the CPU to execute the expected program.

Therefore, a non-volatile memory is needed to store the initial program, so that its contents are retained across power-off and, upon power-on, the CPU can immediately fetch instructions from it. The simplest solution is ROM (Read-Only Memory), from which the content read at the same location is always the same.

There are many ways to implement ROM, but in general, information (in this case also the program) is stored in the ROM through some mechanism, and this storage mechanism is not affected by power loss, thereby providing non-volatile properties. If we consider ease of use within ysyxSoC, the most suitable choice is mask ROM, abbreviated as MROM, whose essence is to "hard-code" the information into the gate circuit, making its access method very direct for the NPC.

However, due to certain problems with MROM, we do not plan to use it during tape-out. Nevertheless, as the first simple non-volatile memory in ysyxSoC for storing programs, MROM is very suitable for testing our integration into ysyxSoC. We have added an MROM controller with an AXI4 interface to ysyxSoC, and its address space is 0x2000_0000~0x2000_0fff.

Test MROM access

Modify the reset PC value of the NPC so that it fetches the first instruction from the MROM, and modify the mrom_read() function so that it always returns an ebreak instruction. If your implementation is correct, the first instruction fetched by the NPC will be ebreak, thus ending the simulation.

Since NEMU does not yet support MROM, and the NPC needs to fetch instructions from MROM at this point, the DiffTest mechanism cannot work correctly for now. However, the current test program is still very small, so you can temporarily disable DiffTest; we will come back to handle the DiffTest issue later.

Output the first character

Once the program can be stored, we need to consider how to output. For this purpose, the SoC also needs to provide a most basic output device. A real SoC usually uses UART16550, which contains some device registers for setting the character length, baud rate, and other information. When the sending queue is not full, characters can be sent by writing to the corresponding device register.

ysyxSoC has already integrated a UART16550 controller. To test it, we first write the simplest program char-test, which directly outputs a character and then falls into an infinite loop:

#define UART_BASE 0x?L
#define UART_TX   ?
void _start() {
  *(volatile char *)(UART_BASE + UART_TX) = 'A';
  *(volatile char *)(UART_BASE + UART_TX) = '\n';
  while (1);
}

Output the first character in ysyxSoC

You need to:

1. According to the device address space convention in ysyxSoC and the address of the output register in the UART manual (in the relevant subdirectory under ysyxSoC/perip/), fill in the ? in the above C code so that the code can correctly access the output register to output a character
2. Compile char-test with the gcc and objcopy commands, and extract the code sections in the ELF file separately into char-test.bin
3. Modify the relevant code of the simulation environment to read char-test.bin and use it as the contents of the MROM, and then correctly implement the mrom_read() function so that it returns the contents at the corresponding location in the MROM according to the parameter addr

If your implementation is correct, the simulation will output a character A to the terminal.

Hint: If you don't know how to achieve the above through the gcc and objcopy commands, you can refer to the video or courseware of a certain OSOC lecture. If you don't know which lecture to refer to, we suggest you go through all the videos and courseware — we believe this will help you fill in many knowledge gaps you may not yet be aware of.

RTFM to understand the bus protocol

If you find the behavior of the bus difficult to understand during simulation, try RTFM first to understand as many details in the manual as possible. As the complexity of the project grows, you will pay an increasingly high price for not RTFMing carefully.

If you inspect the generated ELF file with tools such as objdump, you will find that the address of the code section is near address 0, which is inconsistent with the address space of the MROM. In fact, this program is very small, and we can easily confirm that no matter which address it is placed at, it will execute correctly as expected. For more complex programs, the above condition may not hold, and we need to explicitly link the program to a correct location so that the NPC can execute the program correctly after reset. We will solve this problem later.

In addition, in a real hardware scenario, the serial port also needs to convert characters into serial output signals according to the baud rate and transmit them through wires to the receiving end of the serial port. Therefore, before sending characters, the software also needs to set the correct divisor in the serial port's configuration register. However, there is no receiving end of the serial port in the current ysyxSoC simulation environment, so we have added some print statements in the RTL code of the serial port controller to directly print the characters in the serial port's sending queue, and thus the software does not need to set the divisor. Consequently, the above code may not work properly in a real hardware scenario, but as an early test, this makes it convenient for us to quickly check whether characters are correctly written into the serial port's sending queue. After we successfully run enough programs, we will add the divisor setting so that the code can work in a real hardware scenario.

Output even if line breaks are removed

The above char-test also outputs a newline character after outputting the character A. Try outputting only the character A without the newline; you should observe that the simulation does not even output the character A. But if you output a newline after every character, the printed information will be hard to read.

To solve this problem, you only need to pass an option to verilator. Try to find and add this option through RTFM based on your understanding of the problem. If you add the correct option, you will see that even the above program, which only outputs a single character A, can output successfully.

Hint: The PA lecture notes have already discussed related issues in several places. If you have no recollection of them, we suggest you carefully reread every detail of the notes to identify and fill in the gaps.

A more practical SoC

After confirming that ysyxSoC can output a character, we believe that the data path for the NPC to access devices is basically established. However, although MROM can store programs well, it has a big problem: it does not support write operations. But most programs need to write data to memory. For example, the C calling convention allows the called function to create a stack frame on the stack and access data through it. Therefore, an SoC that contains only MROM as its memory may not be able to support programs that need to call functions, which is clearly impractical. To support write operations, we need to add RAM as memory and allocate the program's data in RAM.

The simplest RAM is the SRAM we mentioned earlier, and we can integrate SRAM memory into the SoC. SRAM can be manufactured with the same process used to make processors, and its read/write latency is only 1 cycle, so it is very fast. However, SRAM has a low storage density and occupies a certain chip area, so in terms of tape-out cost it is very expensive. Considering the tape-out cost, we only provide 8KB of SRAM in the SoC. We have added an SRAM controller with an AXI4 interface to ysyxSoC, whose address space is 0x0f00_0000~0x0f00_1fff. Note that in the earlier introduction, the SRAM address space is 0x0f00_0000~0x0fff_ffff, a total of 16MB; this only means that ysyxSoC reserves 16MB of address space for SRAM, but considering the actual cost, only 8KB of it is used. The remaining address space is unused, and the NPC should not access it.

With this part of SRAM space, we can consider allocating the stack in the SRAM space, thereby supporting the execution of some AM programs.

Add the AM runtime environment for ysyxSoC

In order to run more programs, we need to provide the corresponding runtime environment for programs based on ysyxSoC. Oh, isn't this just implementing a new AM? This should already be familiar to you. However, we still need to consider the impact that some attributes of ysyxSoC have on the runtime environment.

First, let's look at TRM. Reviewing the contents of TRM, we need to consider how to implement TRM's API on ysyxSoC:

• A memory area that can be freely used for computation - the heap area
  • The heap area needs to be allocated in a writable memory area, so it can be allocated in SRAM
• The program "entry" - main(const char *args)
  • The main() function is provided by the program running on AM, but we need to consider the entry of the entire runtime environment, i.e., we need to link the program into the MROM address space and ensure that the first instruction of TRM is consistent with the PC value after the NPC resets
• The way to "exit" a program - halt()
  • ysyxSoC does not support functions such as "shutdown". For convenience, we can use the ebreak instruction to let the simulation environment end the simulation
• Print characters - putch()
  • Can be output through the UART16550 in ysyxSoC

Since the NPC starts executing from the MROM after reset, and the MROM does not support write operations, we need to pay extra attention to the following:

1. The program must not contain write operations to global variables
2. The stack area needs to be allocated in the writable SRAM

Add the AM runtime environment for ysyxSoC

Add a new AM riscv32e-ysyxsoc, and provide TRM's API as described above. After adding it, compile the dummy test in cpu-tests to riscv32e-ysyxsoc, and try to run it in ysyxSoC's simulation environment.

Hint: To complete this task, you need some knowledge of linking. If you are not familiar with it, you can refer to the related videos and courseware of OSOC.

Tests that cannot run

Try running fib in cpu-tests on ysyxSoC, and you will find that the run fails. Try reading the prompt message; how do you think this problem should be solved?

Re-add DiffTest

We have added MROM and SRAM, and for some time to come we will be running programs on MROM and SRAM. But currently NEMU does not have MROM and SRAM. If we skip accesses to MROM and SRAM during DiffTest, we would skip the execution of all instructions, making DiffTest unable to serve its intended purpose.

To re-add DiffTest, you need to add MROM and SRAM to NEMU, and when initializing DiffTest in the NPC's simulation environment, synchronize the contents of the MROM to NEMU, and then check every instruction executed in the MROM.

You can modify NEMU's code as you see fit, but we still recommend that you avoid adding new DiffTest APIs as much as possible; the DiffTest APIs provided by the framework code are already sufficient to implement the above functionality.

Make the NPC throw an Access Fault exception

Although this is not required, we suggest that you add the implementation of Access Fault to the NPC. When an unexpected access to an unallocated address space occurs, or a device returns an error, ysyxSoC can convey the relevant error information through the AXI resp signal. Even if the program has not started CTE, you can make the NPC jump to address 0 when such events occur, making you feel that the program is not running properly. Compared with letting the NPC continue running while ignoring these error events, this may save you a lot of debugging time.

Memory access test

Once the dummy test can be executed, we believe that the NPC can basically access ysyxSoC's SRAM successfully. We know that memory access is the foundation of program execution. To test the memory access behavior more thoroughly, we need to write a program mem-test to test a larger range of memory.

In terms of scope, mem-test hopes to test all writable memory areas. However, mem-test itself requires the support of the stack area for its execution, and the stack area needs to be allocated in a writable memory area, so the stack area must be bypassed during the test to avoid its contents being overwritten, which would cause mem-test itself to malfunction. We can place the stack area at the end of SRAM, set the start address of the heap area at the beginning of SRAM, and set the end address of the heap area at the start address of the stack area (i.e., the initial value of the stack top). After setting up the range of the heap area, we can use the heap area as the test range of mem-test.

In terms of test method, we adopt the most intuitive approach: first write some data into the memory area, then read it back and check it. We can make the written data related to the memory address to facilitate the check, for example data = addr & len_mask. The following diagram illustrates the relationship between the written data and the address for 8-bit, 16-bit, 32-bit, and 64-bit accesses.

    SRAM_BASE                                    SRAM_BASE + 0x10
        |                                               |
        V                                               V
        +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
8-bit   |00|01|02|03|04|05|06|07|08|09|0a|0b|0c|0d|0e|0f|
        +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
16-bit  |00|00|02|00|04|00|06|00|08|00|0a|00|0c|00|0e|00|
        +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
32-bit  |00|00|00|0f|04|00|00|0f|08|00|00|0f|0c|00|00|0f|
        +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
64-bit  |00|00|00|0f|00|00|00|00|08|00|00|0f|00|00|00|00|
        +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+

The test consists of two steps: the first step is to write the corresponding data into each memory area in turn, and the second step is to read the data from each memory area in turn and check whether it matches the data written previously. The above process can be repeated with 8-bit, 16-bit, 32-bit, and 64-bit write modes.

Test memory access via mem-test

Write a new program mem-test in am-kernels to complete the above memory test functionality. If an inconsistency is found when checking the data, end mem-test's execution through halt().

Some hints:

1. Currently, the program's global variables are allocated in the MROM, so your program cannot contain write operations to global variables
2. The code of printf() is relatively complex. Calling printf() may make the program size exceed the MROM space, and it also contains quite a few memory access operations, possibly even including writes to global variables. Therefore, we currently do not recommend using printf() to print information; however, DiffTest, trace, and waveforms should be sufficient to help you debug
3. To avoid the impact of compilation optimization, you need to find a way to confirm that the program actually performs the expected memory access operations during execution

Intelligent linking process

You have already implemented printf() in klib, but if printf() is not called in mem-test, the linked executable indeed does not contain printf()'s code. This intelligent linking method can avoid generating unnecessary code when the memory space is limited.

Do you know which steps or options in the current linking process accomplish the above functionality?

Real memory access test programs

In fact, the above test method cannot comprehensively test various memory access problems. Real memory test programs usually use more complex patterns to test memory reads and writes, which can cover multiple faults, such as memtest86. There are also memory test units implemented in hardware that can perform deeper tests; interested students can learn about MBIST (Memory Build-In Self Test).

Support writing operations of global variables

Many programs write to global variables, so we also need to find a solution to support write operations to global variables. Since global variables are located in the data segment, for ease of description, "data segment" will be used below to refer to "global variables". A straightforward idea is that, since the MROM does not support write operations, we allocate the data segment in SRAM. However, at system startup, SRAM does not contain valid data, so the data segment can only be placed in the MROM to be accessible at startup. To solve this problem, we can load the data segment from the MROM into SRAM before the program actually starts executing, and let the subsequent code access the data segment loaded into SRAM, thereby supporting write operations to global variables through SRAM's writable property.

In fact, a real operating system also needs to load programs into memory for execution, which involves many complex operations. Therefore, the code that performs the above loading operation can also be regarded as a loader, except that its functionality is still very simple at present: it is only responsible for loading the program's data segment from the MROM into SRAM. But since this loader works at system startup, we call it the bootloader.

In simple terms, we need to implement the following three points:

1. Obtain the address MA (mrom address) of the data segment in the MROM, the address SA (sram address) in SRAM, and the length LEN of the data segment
2. Copy the data segment from MA to SA
3. Let the program code access the data segment through SA

For point 2, we only need to call memcpy() to implement it. For MA, we can define a symbol before the start of the data segment in the linker script, so that the bootloader can obtain the address of this symbol at runtime; for LEN, we define a symbol after the end of the data segment in the linker script and subtract it from the above symbol. One issue that needs to be considered is how to obtain SA. On the one hand, since SA is an address, and the addresses in a program can only be determined during the relocation stage of linking, SA can be determined at the earliest only at link time. On the other hand, since point 3 requires the subsequent code to access the data segment through SA, and it is very difficult for the bootloader to modify the access addresses in the corresponding instructions at runtime, SA must be determined before execution. Considering both points together, we can conclude that SA can only be determined at link time, and we need to define SA in the linker script.

For this purpose, we need to use two kinds of symbolic addresses in the linker script. One is the virtual memory address (VMA), which indicates the address where an object resides while the program is running; the other is the load memory address (LMA), which indicates the address where an object resides before the program runs. Normally, these two addresses are the same. But in the above requirements, they differ: the data segment is stored in the MROM, but the program code needs to access the data segment loaded into SRAM, i.e., MA is the LMA and SA is the VMA.

To distinguish between the two kinds of addresses, we need to make slight modifications to the linker script. First, we need to define two memory regions:

MEMORY {
  mrom : ORIGIN = 0x20000000, LENGTH = 4K
  sram : ORIGIN = 0x0f000000, LENGTH = 8K
}

Then, when describing the mapping between sections and memory regions, explicitly state the VMA and LMA of each section. For example:

SECTIONS {
  . = ORIGIN(mrom);
  .text : {
      /* ... */
  } > mrom AT> mrom
  /* ... */
}

Here, what follows > indicates the memory region where the VMA is located, and what follows AT> indicates the memory region where the LMA is located. The above linker script means that the code segment is linked into the MROM space and is also located in the MROM space.

Load the data segment into memory through the bootloader

According to the above content, load the data segment into SRAM before TRM calls the main() function, thereby supporting the subsequent code's writes to global variables. If your implementation is correct, you should be able to run all tests in cpu-tests except hello-str.

Some hints:

1. The loading process of the bootloader can be implemented by writing some simple loops in assembly code, or by calling functions such as memcpy() in C code
2. For writing the linker script, you can refer to the official documentation
   • In order to force everyone to RTFM carefully, we have ignored a small detail in the above introduction that you may have thought of. This detail has already been mentioned in the official documentation, and even if you really didn't think of it, you can learn it by reading the documentation carefully
3. You can use the --print-map option to see how ld performs the linking

Output via serial port

After supporting writes to global variables, any computable program that fits in the MROM and SRAM can theoretically run. Finally, let's discuss the implementation of putch().

Implement putch()

Imitating the char-test above, implement the putch() functionality by writing characters to the UART16550. After implementation, run the hello program. If your implementation is correct, you will see the NPC output several characters.

However, you will find that the NPC does not output all the characters. Although this is not what we expect, it is the expected behavior for now, and we will fix this problem next.

Observe the behavior of the NPC output

Try modifying the code of hello.c to increase or decrease the length of the string, and observe the behavior of the NPC outputting the string. Based on your observations, what do you guess the cause might be?

A more essential way to phrase this question is: do you understand every detail of "a program outputting a character in ysyxSoC"? Although we will reveal the answer next, students willing to take on the challenge can pause reading and try RTFSC to trace through every detail; after all, arriving at the answer through your own exploration brings a greater sense of accomplishment.

You may have observed this strange phenomenon: the NPC outputs some characters and also successfully ends the simulation through the ebreak instruction, indicating that the program itself has no fatal problems, but some characters have disappeared: the program should have written them to the serial port, yet they cannot be seen on the terminal. If you observe carefully, you will find that no matter how long the string in the program is, the terminal outputs at most 16 characters. Since 16 is a power of 2, it hardly seems a coincidence, and it likely hints at some kind of configuration.

Of course, no matter how brilliant a guess is, it ultimately has to be verified by RTFSC, so here we still leave the RTFSC process to you. After careful RTFSC, you will find that the cause of the above problem is simply that the software did not initialize the serial port! Because of the missing initialization, the sending function of the serial port does not work, so the characters written to the serial port keep occupying its sending queue, and once the queue is full, no more characters can be written.

In fact, before outputting characters to the serial port, the software needs to perform the following initialization:

1. Set the serial port's transceiver parameters, including the baud rate, character length, whether a parity bit is used, the width of the stop bits, etc.
   • The baud rate refers to the number of characters transmitted per second. However, the baud rate is usually not set directly in the register; instead, a divisor inversely proportional to the baud rate is set: the smaller the divisor, the higher the baud rate and the faster the transmission rate, but due to electrical characteristics, the bit error rate also increases and the probability of successful character transmission decreases; conversely, the larger the divisor, the lower the baud rate, the slower the transmission rate, and the longer the software has to wait. The value of the divisor is also related to the working frequency of the serial port controller, where the latter refers to the number of bits transmitted per second by the serial port; you can RTFM to learn the specific relationship between the two.
   • The parameter configurations of the sending and receiving ends of the serial port must be exactly the same to send and receive characters correctly. A set of parameter configurations is usually described in a form such as 115200 8N1, which means a baud rate of 115200, a character length of 8 bits, no parity bit, and 1 stop bit.
2. Set interrupts as needed; however, the NPC currently does not support interrupts, so you may skip this

Correctly implement serial port initialization

You need to add code to TRM to set the serial port's divisor register. Since ysyxSoC is essentially still a simulation environment, with no serial port receiving end and no concept of electrical characteristics, you can currently set the above divisor arbitrarily without worrying about the bit error rate. Of course, in a real chip, the setting of the divisor register needs careful consideration. In addition, we will connect a serial port terminal in the subsequent lab content for more tests.

As for how exactly to set the divisor, you can RTFM to understand the functionality of the UART IP, or RTFSC, combining it with the RTL implementation of the UART16550 registers, to help you understand how the set divisor works.

If you set the divisor register small enough, you will observe that the hello program outputs a few more characters, but characters are still lost. To solve this problem, we need to ensure that the sending queue must have a free slot to write to before writing a character to the serial port. This can be achieved by querying the serial port's status register: the software can poll the relevant register until it is sure that the written characters will not be lost.

Poll the serial port's status register before outputting

You need to modify the code of putch() to query the status of the serial port's sending queue before outputting. As for how to query, similarly, you can RTFM to understand the functionality of the UART IP, or RTFSC, combining it with the RTL implementation of the UART16550 registers, to help you understand the related functionality.

Once the serial port works correctly, TRM can run more programs. However, the scale of programs is still limited by the sizes of the MROM and SRAM. Affected by the manufacturing process, if we want to use larger memory at an acceptable cost, we need to use some slower memories.

Reprogrammable non-volatile memory

First, let's solve the problem of program storage. In addition to its high cost, another drawback of the MROM is its weak programmability. In fact, the MROM only supports programming at manufacturing time, i.e., its contents are decided during RTL design; after back-end physical design, the wafer fab will fabricate the mask used for photolithography according to the layout, and then use this mask to manufacture the chip — this is also the meaning of "mask" in mask ROM. But once the chip is manufactured, the contents stored in the MROM cannot be changed. If the program running on the chip had to be replaced through a re-tape-out, the cost would be unacceptable.

With the development of storage technology, people invented ROMs that can be reprogrammed and erased, and the widely used flash memory is one of them. Users can erase the contents stored in a flash memory and rewrite them under certain conditions. Generally speaking, users only need to buy a burner worth a few dozen yuan, and then update the contents of the flash through the burning software. In this way, the cost of replacing the program stored in the flash becomes acceptable.

The popularity of USB flash drives even removes the need to buy a dedicated burner. A USB flash drive is essentially a flash memory with a USB interface, plus an MCU. Today's operating systems all have built-in flash drivers for the USB protocol, so users only need to plug the USB flash drive into a computer to write data to it. However, this is a bit too complex for the current NPC: it would not only require a USB controller, but also the corresponding driver to be run to complete the burning operation. Therefore, OSOC still adopts the burner solution.

Flash storage unit

Under construction

There are no programming tasks in this section. Interested students can read the courseware or watch the bilibili recording first.

Internal structure of flash chips

To help everyone further understand flash memory, we introduce the internal structure of the W25Q128JV flash chip. This flash chip has 24 address lines and can store 16MB of data, which is enough to hold most of our test programs. The entire storage array of the flash chip is divided into 256 Blocks, each 64KB in size; each block is further divided into 16 Sectors, each 4KB in size; and each sector is further divided into 16 Pages, each 256B in size.

In a flash chip, the byte is the smallest unit of reading and supports random reads. Write operations are more complicated: to write 0, you only need to program the corresponding storage cell; but to write 1, you must first perform an erase operation. Due to the physical characteristics of flash storage cells, the sector is the smallest unit of erasure, i.e., we need to read out all the contents of the sector, erase that sector, and then program it according to the new data. As you can see, the write overhead of flash is much larger than the read overhead.

In addition to the storage array, a flash chip also contains several registers, including some address registers used to control the read/write address of the flash chip, control registers used to control the behavior of the chip (such as write protection, access permissions, etc.), and status registers used to hold the current read/write status of the flash chip. As you can see, the inside of a flash chip is essentially a device controller!

To access a flash chip, the outside must send commands to it. After receiving an external command, the flash chip parses it and then executes the specific function of the command. This is very similar to the process of a CPU executing instructions: the CPU's instruction cycle includes fetch, decode, execute, and PC update, while for most devices, including flash chips, the processing process includes receiving a command, parsing, executing, and waiting for a new command. As for the format and functionality of the commands, they are of course defined by the corresponding manual. For example, the 8-bit command 03h means reading data from the flash chip, and the command is followed by a 24-bit address of the storage cell. Therefore, if you have learned CPU design, you are fully capable of designing the core logic of a flash chip according to its manual.

With the help of the bus, we can easily translate the CPU's memory access requests into read/write commands for the flash chip. Taking a load request whose target address is in the flash space as an example, when the CPU's LSU executes a load instruction, it initiates a read transaction on the bus. This read transaction passes through the Xbar and finally reaches the flash controller. The flash controller checks the attributes of the transaction, finds that it is a read transaction, then generates the corresponding read command, generates the address of the read command according to the address in the bus transaction, and sends this command to the flash chip. After a while, the flash controller obtains the data read from the flash chip and transmits it to the CPU as the response to the bus transaction. After the CPU's LSU receives the read result, the load instruction continues to execute.

RTFSC to understand the process of reading data from flash

ysyxSoC contains the code implementation of the above process and maps the flash storage space to the CPU's address space 0x3000_0000~0x3fff_ffff. You need to first define the macro FAST_FLASH in ysyxSoC/perip/spi/rtl/spi_top_apb.v, and then try to understand the above process in combination with the code.

As for writing to flash, since writing to flash requires first erasing an entire sector and then rewriting the whole set of data, multiple commands need to be sent to the flash chip. However, at present we only intend to use flash to replace the MROM, so that the NPC can fetch valid instructions from flash upon reset. Therefore, we will not perform write operations on the flash chip for the time being, and ysyxSoC's flash chip code only supports read operations.

Read data from flash

After understanding the process of reading data from flash, you can now test this process with code:

1. Define an array representing the flash storage space in the simulation environment
2. Write some contents into the above array when the simulation environment initializes
   • This operation can be regarded as simulating the process of burning data into the flash chip
3. Correctly implement the flash_read() function so that it returns the contents at the corresponding location in flash according to the parameter addr
4. Write a simple test program in am-kernels to read out the contents from the flash storage space and check whether they match the contents set when the simulation environment initialized

Access flash chips through the SPI bus protocol

Because the manufacturing process of flash chips differs from that of processors, the processor chip and the flash chip chip must be manufactured separately and then soldered onto the board, communicating through the traces on the board. For this reason, the number of pins becomes a consideration. On the one hand, too many pins make it hard to keep the chip small, which negatively affects the layout and area of the board; on the other hand, the traces on the board are usually longer than those inside the chip, and signals are more susceptible to interference, so too many pins also cause dense routing on the board, and these traces can easily interfere with each other, affecting signal stability.

Taking the read command mentioned above as an example, a single read operation involves at least an 8-bit command, a 24-bit address, and 8-bit data, which already occupies 40 bits of signals. Therefore, routing all these signals to the outside of the flash chip through pins is not a good solution.

To reduce the number of pins of the flash chip, an SPI bus interface is usually added to the flash chip. SPI stands for Serial Peripheral Interface, a serial bus protocol that communicates between a master and a slave through a few signal lines.

The SPI bus has only 4 kinds of signals in total:

1. SCK - the clock signal sent by the master, only 1 bit
2. SS - slave select, the selection signal sent by the master, used to specify the communication target; each slave corresponds to 1 bit
3. MOSI - master output slave input, the data line through which the master communicates with the slave, only 1 bit
4. MISO - master input slave output, the data line through which the slave communicates with the master, only 1 bit

To communicate over the SPI bus, the master usually first selects the target slave with the SS signal, then sends SPI clock pulses to the slave through the SCK signal while converting the information to be sent into a serial signal, transmitting it to the slave bit by bit through the MOSI signal; then it monitors the MISO signal and converts the serial signal received through MISO back into parallel information, thereby obtaining the slave's response.

The slave works in a similar way: if the slave receives SCK clock pulses while the SS signal is active, it monitors the MOSI signal and converts the serial signal received through MOSI back into parallel information, thereby obtaining the command sent by the master; after processing the command, it converts the response information into a serial signal and transmits it to the master bit by bit through the MISO signal.

As we can see, apart from the state machine, the core of implementing the SPI bus protocol is the conversion between serial and parallel signals, specifically how the sender sends signals and how the receiver samples and receives them. On the one hand, the endianness of transmission needs to be considered: whether to send from the most significant bit to the least significant bit, or vice versa; on the other hand, the timing of sending and sampling (clock phase) needs to be considered: whether to send/sample on the rising edge or the falling edge of SCK. Sometimes the level of the clock when idle (clock polarity) is also agreed upon: whether it idles high or low. During the sending and sampling process, SCK plays the role of synchronization: both parties jointly agree on the endianness and timing of sending/sampling and correctly implement the agreement at the RTL level. In real scenarios, different slaves may have different agreements, which means that when communicating with different slaves, the master needs to adapt its sending and sampling to the slave's agreement.

However, upper-level software does not want to care about these signal-level behaviors, so the SPI master also needs to abstract these signal-level behaviors into device registers; the upper-level software only needs to access these device registers to query or control the SPI master. In fact, in the bus architecture, the SPI master has a dual identity: on the one hand, it is a slave on the AXI (or other AMBA protocol, such as APB) bus, responsible for receiving commands from the CPU; on the other hand, it is the master on the SPI side, responsible for sending commands to SPI slaves. Therefore, we can also regard the SPI master as a bridge module between AXI and SPI, used to convert AXI requests into SPI requests, thereby communicating with SPI slaves.

ysyxSoC integrates an implementation of the SPI master and maps its device registers to the CPU's address space 0x1000_1000~0x1000_1fff; the relevant code and manuals are in the corresponding subdirectories under ysyxSoC/perip/spi/. With the abstraction of device registers, we can sort out the behavior of the upper-level software. To communicate with different slaves, the master generally needs to support multiple conventions, and which convention to use is configured through the device registers. Before communicating with a slave, the SPI driver first sets the SS register to select the target slave, configures the control registers of the SPI master according to the slave's convention, then writes the data to be transmitted into the transmit data register, and finally writes a command meaning "start transfer" into a control register. The SPI driver can poll the status register of the SPI master: when the status flag is "busy", it waits, and only when the status flag becomes "idle" can it read the slave's response from the receive data register.

Implement the bit flip module based on the SPI protocol

To become familiar with and test the basic SPI flow, let's write a simple bit flip module, bitrev. This module takes an 8-bit data input and outputs the result of flipping the bits of that data, i.e., swapping bit 0 with bit 7, bit 1 with bit 6, and so on.

Specifically, if you choose Verilog, you need to implement the corresponding code in ysyxSoC/perip/bitrev/bitrev.v; if you choose Chisel, you need to implement the corresponding code in the bitrevChisel module of ysyxSoC/src/device/BitRev.scala, and modify Module(new bitrev) in ysyxSoC/src/SoC.scala to instantiate the bitrevChisel module.

If the input and output signals of this bitrev module were 8 bits, it would be a simple digital circuit assignment; but since the bitrev module communicates through the SPI bus, you also need to implement the serial/parallel signal conversion (which is not difficult either — our Chisel reference code only requires adding 5 lines). Some hints are as follows:

1. The SS signal output by the SPI master is active low, and the slave is required to drive the MISO signal high when idle
2. Since SCK only generates pulses during SPI transfers, you may need to use asynchronous reset. However, since this bitrev module does not participate in tape-out, using asynchronous reset in it does not affect the tape-out flow
   • If you use Chisel, you can refer to the explanation about Reset in the Chisel documentation
3. If you use Chisel and want to trigger on the falling edge of the clock, you can refer to this post
4. You also need to cancel the macro FAST_FLASH defined in ysyxSoC/perip/spi/rtl/spi_top_apb.v, so that APB requests can access the device registers of the SPI master

After implementing the bitrev module in hardware, you also need to write a program to test it. Try writing an AM program that drives the SPI master to input an 8-bit data to the bitrev module, then reads out the processed result and checks whether it matches the expectation. Specifically:

1. Set the data to be sent into the TX register of the SPI master
2. Set the divisor register, which indicates the ratio between the SCK frequency and the current clock frequency of the SPI master during transmission. Since verilator has no concept of frequency, you can set a divisor that makes the SCK frequency as high as possible
   • In a real chip, an excessively high SCK frequency may prevent the slave from working correctly, so the divisor register must be set to meet the slave's working frequency requirement
3. Set the SS register to select the bitrev module as the slave
   • ysyxSoC has already connected bitrev as an SPI slave to the SPI master, and its slave number is 7
4. Set the control register. Specifically, you need to set each field of it properly; the descriptions of some fields are as follows:
   • CHAR_LEN - since the lengths of both the input data and the output data are 8 bits, the transfer length should be 16 bits
   • Rx_NEG, Tx_NEG, and LSB - since bitrev is only a test module and does not participate in the final tape-out, we do not specify these details of the bitrev module; the convention is left to you. You need to choose a convention, and then implement the bitrev module, set the SPI control register, and write the software according to this convention, so that the three can communicate under the same convention
   • IE - we do not use the interrupt function for now
   • ASS - whether to set it or not is up to you, but it needs to be considered together with the software
5. Poll the completion flag in the control register until the SPI master completes the data transfer
6. Read the data returned by the slave from the RX register of the SPI master

We give an example of one data transfer. Note that this is only an illustrative diagram, and your implementation does not need to be exactly the same:

                +---+   +---+   +---+   +---+   +---+   +---+   +---+   +---+   +---+   +---+   +---+   +---+   +---+   +---+   +---+   +---+
 SCK            |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |
        --------+   +---+   +---+   +---+   +---+   +---+   +---+   +---+   +---+   +---+   +---+   +---+   +---+   +---+   +---+   +---+   +------------
        --------+                                                                                                                               +--------
 SS             |                                                                                                                               |
                +-------------------------------------------------------------------------------------------------------------------------------+
        +-------+-------+-------+-------+-------+-------+-------+-------+-------+------------------------------------------------------------------------
MOSI            |   b7  |   b6  |   b5  |   b4  |   b3  |   b2  |   b1  |   b0  |
                +-------+-------+-------+-------+-------+-------+-------+-------+
        +-----------------------------------------------------------------------+-------+-------+-------+-------+-------+-------+-------+-------+--------
MISO                                                                            |   b0  |   b1  |   b2  |   b3  |   b4  |   b5  |   b6  |   b7  |
                                                                                +-------+-------+-------+-------+-------+-------+-------+-------+

The SPI transfer process involves many details, and you will most likely need RTFM and RTFSC to help you understand them.

Read data from flash through the SPI bus

Try writing an AM program that implements a function with the prototype uint32_t flash_read(uint32_t addr). Note that this flash_read() function is different from the DPI-C interface function of the same name mentioned earlier. This flash_read() function reads the 32-bit contents starting at address addr in the flash chip by driving the SPI master. This process is similar to the bitrev example above:

1. Set the command to be sent to the flash chip into the TX register of the SPI master
2. Set the divisor register
3. Set the SS register to select the flash chip as the slave; its slave number is 0
4. Set the control register:
   • CHAR_LEN - since the read command is 32 bits long in total and 32 bits of data need to be read out, the transfer length should be 64 bits
   • Rx_NEG and Tx_NEG - need to be set according to the slave's relevant documentation
     • In a real chip, incorrectly setting Rx_NEG and Tx_NEG may cause hold time violations during circuit operation, making it impossible to sample the correct data. However, verilator has no concept of timing, so some incorrect settings may still produce correct results; nevertheless, we still recommend that you RTFM and then set them strictly according to the conventions
   • LSB - needs to be set according to the slave's relevant documentation; if necessary, the endianness of the read data can be adjusted in software
   • IE - we do not use the interrupt function for now
   • ASS - whether to set it or not is up to you, but it needs to be considered together with the software
5. Poll the completion flag in the control register until the SPI master completes the data transfer
6. Read the data returned by the slave from the RX register of the SPI master

After implementing flash_read(), use this function to read out the contents from the flash storage space and check whether they match the contents set when the simulation environment initialized.

Load a program from flash and execute it

Try storing the char-test program mentioned above into the flash chip, write a test program that uses flash_read() to read char-test from flash into some address in SRAM, and then jump to that address to execute char-test.

Fetch instructions from flash

After being able to read data from flash correctly, we can consider putting the program to be executed into flash and letting the NPC fetch its first instruction from flash.

Wait, something seems off... We just read the contents from flash through the function flash_read(). This function is also part of the program; it is compiled into a sequence of instructions placed in the MROM, and the NPC fetches and executes instructions from the MROM. But if the instruction sequence of flash_read() is also burned into flash, who will fetch and execute the instructions of flash_read()? This becomes a "chicken and egg" circular dependency problem.

Looking deeper, the root of this problem is that we are trying to implement instruction fetching, which is a hardware-level behavior, through a software function. This is unreasonable, because instruction fetching is a hardware-level action. Therefore, we should try to implement the functionality of "fetching instructions from flash" at the hardware level.

To implement the functionality of flash_read() at the hardware level means accessing the registers of the SPI master in a certain order on the hardware. You may have thought of it: implement the functionality of flash_read() with a state machine! Unlike the earlier approach of loading the program from flash and then executing it, this way of fetching instructions from the flash chip does not require reading the program into memory (the SRAM above) before executing it, so it is also called "eXecute In Place" (XIP).

To distinguish it from the normal access to the SPI master, we need to map the functionality of accessing flash through XIP to an address space different from that of the SPI master's device registers. In fact, we can slightly adjust the previously accessed flash storage space 0x3000_0000~0x3fff_ffff and define it as the flash storage space accessed through XIP. The Xbar in ysyxSoC has already mapped both the address space of the SPI master 0x1000_1000~0x1000_1fff and the flash storage space 0x3000_0000~0x3fff_ffff to the APB port in the ysyxSoC/perip/spi/rtl/spi_top_apb.v module, i.e., the APB port in spi_top_apb.v can receive requests to both of the above address ranges, and you can distinguish them by checking the target address of the APB request.

Access flash through XIP

To summarize, the general process of implementing XIP is as follows:

1. Check the target address of the APB request; if the target address falls within the address space of the SPI master, access normally and respond
2. If the target address falls within the flash storage space, enter XIP mode. In XIP mode, the input signals of the SPI master are determined by the corresponding state machine
   • The state machine writes the corresponding values into the device registers of the SPI master in turn; the written values are basically the same as those in flash_read()
   • The state machine polls the completion flag of the SPI master and waits for the SPI master to complete the data transfer
   • The state machine reads the data returned by flash from the RX register of the SPI master, processes it and returns it through APB, then exits XIP mode

Specifically, if you choose Verilog, you need to implement the corresponding code in ysyxSoC/perip/spi/rtl/spi_top_apb.v; if you choose Chisel, you need to implement the corresponding code in the Impl class of ysyxSoC/src/device/SPI.scala.

Before fetching instructions through XIP, let's first test whether read requests issued by the CPU can be completed through XIP. Write a test program that directly reads out the contents from the flash storage space through a pointer and checks whether they match the contents set when the simulation environment initialized.

Likewise, we currently do not consider supporting write operations to flash through XIP, so you had better find a way to report an error when a write operation is detected, to help you diagnose the cause of the problem in time.

Execute a program in flash through XIP

Store the char-test program mentioned above into the flash chip, write a test program, and jump into flash to execute char-test.

Replace the MROM with flash

After confirming that instructions can be fetched and executed from flash, we can completely replace the MROM with flash to store the first program. Modify the reset value of the PC so that the NPC fetches its first instruction from flash after reset. You will also need to make a series of modifications to adapt to this change, including... well, that's left for you to figure out, which also tests whether you understand every detail of this process.

Since the size of flash is much larger than the MROM used previously, we can store and execute larger programs now; try running programs that include printf(), such as coremark. If you run microbench, you will find that quite a few subtests cannot run because the heap area is too small.

coremark takes a long time to run

If you were allowed to make changes, how would you reduce coremark's running time?

Hint: RTFSC

Try executing the flash_read() function on flash

You may find errors; try analyzing why they occur.

Add the student ID CSR and output the student ID

If you have already implemented the student ID CSR before, you can skip this task.

To distinguish the NPCs of different students, you can set your own student ID in the identification registers of the CSR. Specifically, you can add the following two CSRs to the NPC:

• mvendorid - reading from it returns the ASCII code of ysyx, i.e., 0x79737978
• marchid - reading from it returns the decimal representation of the numeric part of the student ID; assuming your student ID is ysyx_22068888, it reads 22068888, i.e., 0x150be98

After implementation, you can read out the values of the above two CSRs and output them before the TRM of riscv32e-ysyxsoc enters the main() function.

Wait for the SPI master to complete the transfer through interrupts

Our XIP implementation above continuously polls to check whether the SPI master's transfer is complete. In fact, the SPI master also supports an interrupt notification mode: after setting the IE bit of the control register, the SPI master will issue an interrupt signal when the transfer ends, and the state machine in XIP mode can wait for the SPI master to complete the transfer by waiting for this interrupt signal. Students interested in this can implement the above interrupt-based waiting approach.

Although this does not bring a significant performance improvement, in a real system, implementing interrupt-based waiting can save energy, because the requests and replies issued by polling do not make an actual contribution to the system's operation.

Random access memory with higher storage density

After using flash to solve the problem that the MROM can only store small programs, we still need to consider data storage. If a program needs to write more data than the 8KB that SRAM can provide, then it still cannot run on ysyxSoC. For this, we need a larger memory that can also support the CPU executing store instructions.

DRAM storage unit

DRAM (Dynamic Random Access Memory) is a widely used type of memory nowadays. Compared with SRAM, DRAM has the characteristics of large capacity and low cost. The DRAM storage cell stores 1 bit with one transistor and one capacitor, where the transistor acts as a switch, functionally equivalent to a read/write enable; the capacitor is used to store the 1 bit of information. When the charge in the capacitor exceeds a certain threshold, it is considered 1, otherwise it is considered 0. As we can see, DRAM stores information through electrical properties, so it is a volatile memory; after power-off, all information stored in DRAM is lost. Conversely, when the system is powered on, there is no valid data in DRAM either.

However, capacitors have the property of leakage. If nothing is done, the charge in the capacitor will keep decreasing, and eventually 1 will become 0, making it impossible to tell whether the originally stored data was 1 or 0, resulting in data loss. To avoid this situation, the DRAM storage cells must be refreshed periodically: the DRAM controller reads out the information stored in each storage cell, and if it is 1, it rewrites that storage cell. The write operation restores the capacitor of the storage cell to a high-charge state, so that for the following period of time, a 1 can be read out of the storage cell, thereby preserving the stored information.

As we can see, writing to a DRAM storage cell is essentially charging and discharging a capacitor. Therefore, although DRAM write operations are not as fast as SRAM writes, considering factors such as cost and capacity, DRAM write operations are still suitable for supporting the CPU executing store instructions.

If we do not consider the organization of the storage array or the physical process of read/write, functionally the DRAM chip is very similar to the flash chip introduced above: in addition to various registers, it can also accept externally input commands for operation.

Similar to the flash controller, the module that sends commands to a DRAM chip to drive its operation is called a DRAM controller. The DRAM controller needs to translate the bus transaction requests into operation commands for the DRAM chip, and pass this information to the DRAM chip through the memory bus. In addition, the DRAM controller also needs to periodically send refresh commands to the DRAM chip. However, this also increases the design complexity of the DRAM controller: the DRAM controller must accurately calculate the timing at which refreshes are needed; too many refresh operations will reduce the efficiency of executing read/write commands, while too few refresh operations will cause data loss.

PSRAM chips

There is a type of DRAM chip that integrates the refresh logic internally, called PSRAM (Pseudo Static Random Access Memory) chips. The PSRAM controller does not need to implement the refresh functionality, nor does it need to care about the internal structure of the PSRAM chip, so using such chips is very similar to using SRAM: you only need to provide the address, data, and read/write command to access the data in the chip. One example is the PSRAM chip model IS66WVS4M8ALL, which provides 4MB of storage space; for more information, you can refer to the relevant manual.

With PSRAM, we can try to let ysyxSoC provide programs with a larger writable memory area. Similar to accessing flash chips through the SPI protocol, PSRAM chips also provide an SPI interface. However, unlike flash, PSRAM generally serves as the system's memory, so it is necessary to provide a more efficient access method.

Access PSRAM chips through an upgraded version of the SPI bus protocol

In fact, the SPI protocol has some upgraded versions that can improve the communication efficiency between the master and the slave. The basic SPI protocol is full-duplex, i.e., the master and the slave can simultaneously send messages through MOSI and MISO respectively. But usually, the master first sends a command to the slave, the slave can only process it after receiving the command, and then respond with the processing result to the master; this process only requires a half-duplex channel.

The Dual SPI protocol takes advantage of this, using both MOSI and MISO of the basic SPI protocol simultaneously for transmission in one direction, i.e., the Dual SPI protocol can transmit 2 bits unidirectionally within one SCK clock. Since the meanings of MOSI and MISO have changed, in the Dual SPI protocol their names are changed to SIO0 (Serial I/O 0) and SIO1 respectively.

To distinguish from the transmission method of the basic SPI protocol, the slave usually provides different commands for the master to choose which protocol to use for transmission. For example, the W25Q128JV flash chip model mentioned above provides multiple read commands:

• It provides the 03h command, which performs read operations using the basic SPI protocol, with its command, address, and data all transmitted at 1 bit. Usually, the transmission bit widths of the three are denoted by a triplet (command transmission bit width-address transmission bit width-data transmission bit width). For example, the basic SPI protocol is also denoted as (1-1-1). Taking the read of 32 bits of data as an example, the 03h command needs to execute 8 + 24 + 32 = 64 SCK clocks.
• It also provides the 3Bh command, which performs read operations using the Dual SPI protocol; its command and address are transmitted at 1 bit, but the data is transmitted at 2 bits, denoted as (1-1-2). Taking the read of 32 bits of data as an example, the 3Bh command needs to execute 8 + 24 + 32/2 = 48 SCK clocks. However, no matter how many bits the data is transmitted in, reading data from the flash storage array always takes a certain delay, so the 3Bh command also needs to wait an extra 8 SCK clocks before transmitting data, i.e., the 3Bh command needs to execute 8 + 24 + 8 (read delay) + 32/2 = 56 SCK clocks.
• It also provides the BBh command, which performs read operations using the Dual SPI protocol; its command is transmitted at 1 bit, but the address and data are transmitted at 2 bits, denoted as (1-2-2). Taking the read of 32 bits of data as an example, the BBh command needs to execute 8 + 24/2 + 4 (read delay) + 32/2 = 40 SCK clocks.

Furthermore, there is the Quad SPI protocol (abbreviated as QSPI), which adds two new signals, SIO2 and SIO3, allowing 4 bits to be transmitted unidirectionally within one SCK clock. For example, the W25Q128JV flash chip model mentioned above also provides two other read commands based on the QSPI protocol:

• It provides the 6Bh command, with its command and address transmitted at 1 bit, but the data transmitted at 4 bits, denoted as (1-1-4). Taking the read of 32 bits of data as an example, the 6Bh command needs to execute 8 + 24 + 8 (read delay) + 32/4 = 48 SCK clocks.
• It also provides the EBh command, with its command transmitted at 1 bit, but the address and data transmitted at 4 bits, denoted as (1-4-4). Taking the read of 32 bits of data as an example, the EBh command needs to execute 8 + 24/4 + 6 (read delay) + 32/4 = 28 SCK clocks.

However, in the read commands above, no matter how many bits the address and data parts are transmitted in, the command part is always transmitted at 1 bit. This is because the slave only knows which protocol should be used to transmit the subsequent address and data after parsing the command, so the command part still follows the basic SPI protocol and is transmitted bit by bit. In addition, although the above flash chip model supports multiple transmission methods, the SPI master connected to the flash chip can only transmit using the basic SPI protocol, so it cannot issue the other read commands.

ysyxSoC integrates an implementation of the PSRAM controller and maps the PSRAM storage space to the CPU's address space 0x8000_0000~0x9fff_ffff. The code of the PSRAM controller is located in the ysyxSoC/perip/psram/efabless/ directory, and it uses the wishbone bus protocol. We have encapsulated it into the APB bus protocol (see ysyxSoC/perip/psram/psram_top_apb.v) and connected it into the APB Xbar of ysyxSoC. The PSRAM controller translates the received bus transactions into commands sent to the PSRAM chip. We have chosen to simulate the PSRAM chip model IS66WVS4M8ALL, which supports the QSPI protocol, so the PSRAM controller integrated into ysyxSoC can communicate with the PSRAM chip through the QSPI protocol, allowing more efficient commands to be used to access the PSRAM. ysyxSoC has already connected the PSRAM chip to the PSRAM controller, but does not provide code for the PSRAM chip. To use the PSRAM in ysyxSoC, you still need to implement a behavioral model of the PSRAM chip.

Implement the simulation behavioral model of the PSRAM chip

You need to implement the simulation behavioral model of the IS66WVS4M8ALL chip. You only need to implement two commands in SPI Mode, Quad IO Read and Quad IO Write, whose command encodings are EBh and 38h respectively; the PSRAM controller will only send these two commands to the PSRAM chip.

Specifically, if you choose Verilog, you need to implement the corresponding code in ysyxSoC/perip/psram/psram.v; if you choose Chisel, you need to implement the corresponding code in the psramChisel module of ysyxSoC/src/device/PSRAM.scala, and modify Module(new psram) in ysyxSoC/src/SoC.scala to instantiate the psramChisel module.

Some explanations are as follows:

1. The meaning of the port ce_n is the same as SS in the SPI bus protocol; it is active low
2. The port dio is declared as type inout, and together with an output enable signal, it can implement tri-state logic, which can be used for input or output at the same time, enabling half-duplex transmission of signals
   • In the ASIC flow, the tri-state logic cell in the standard cell library needs to be explicitly instantiated, but here we are only testing in the simulation environment, so there is no need to call the standard cell library
   • If you use Verilog, you can refer to the relevant code of qspi_dio in ysyxSoC/perip/psram/psram_top_apb.v
   • If you use Chisel, since Chisel currently does not support operations on the Analog type other than connections, we have already instantiated a TriStateBuf submodule in the framework code, which decomposes dio into din and dout, UInt signals in the two directions, for subsequent use
3. The storage array only needs to be implemented as a two-dimensional array with a word length of 8 bits; there is no need to care about its physical organization, and the focus should be on the QSPI protocol. In addition, since PSRAM is not a non-volatile memory, there is no need to set its contents during simulation environment initialization, so you can directly define the storage array in Verilog code, or access an array defined in C++ code through DPI-C, which is convenient for tracing with mtrace
4. The manual also has a QPI Mode, whose meaning is different from the QSPI mentioned above, and can be ignored for now
5. For details such as endianness and clock phase, you can RTFM by referring to the relevant manual, or RTFSC by referring to the code of the PSRAM controller
   • To correctly implement the communication between the PSRAM controller and the PSRAM chip, you do not need to modify the code of the PSRAM controller

After implementation, test a small segment of PSRAM access (such as 4KB) with mem-test to check whether your implementation is correct. Note that flash now provides a larger storage space for storing programs, so you can call printf() in mem-test to help you output debugging information, especially some information related to the test progress, so that you can know whether the test is proceeding normally.

In fact, there is a protocol called QPI on top of QSPI, which can further improve the transmission efficiency of the command part, i.e., the command, address, and data are all transmitted at 4 bits, denoted as (4-4-4). However, to be compatible with old SPI masters, the slave is generally in basic SPI mode when powered on, communicating through the basic SPI protocol at that time. If the slave supports the QPI protocol, it will provide a command to switch to QPI mode; the master can send this command to switch the slave to QPI mode, and then communicate with it using the QPI protocol.

Use the QPI protocol to access the PSRAM chip

Try adding QPI mode and a command to enter QPI mode to the PSRAM chip, then modify the code of the PSRAM controller so that after reset, it first sends the command to enter QPI mode to the PSRAM chip through circuit logic, and subsequently communicates with the PSRAM chip in QPI mode.

Note that the addition of this functionality is transparent to the upper-layer software; the upper-layer software can improve the efficiency of accessing the PSRAM without any changes.

Run larger programs

With the support of PSRAM, we can try allocating the data segment in PSRAM, thereby supporting the execution of larger programs.

Run microbench on ysyxSoC

Previously, we allocated the data segment and the heap area in the 8KB SRAM, and running microbench requires more than 8KB of memory, so quite a few subtests could not run. After allocating the data segment and the heap area in the 4MB PSRAM, you should be able to see microbench successfully run all the tests at test scale.

We just ran microbench, which only demonstrates from a flow perspective that allocating the data segment in PSRAM is fine; but before running more programs, we had better first test the 4MB PSRAM completely with mem-test. However, completely testing the read/write of this 4MB storage space with mem-test in the simulation environment would take a very long time.

This is because we are currently executing mem-test on flash: fetching one instruction from flash requires at least 64 SCK clock cycles; and since the SPI master generates the SCK clock signal by frequency division, even with the most efficient divide-by-2, the SPI transfer process alone takes at least 128 CPU clock cycles; adding the control overhead of the XIP state machine, fetching one instruction from flash takes about 150 CPU clock cycles in total.

Due to the existence of loops and function calls, most of the code in a program is executed repeatedly. Compared with letting the program keep executing in flash, spending some time beforehand to load the code into a memory with higher access efficiency than flash and then letting the program execute repeatedly in the latter can effectively improve the program's execution efficiency. However, this loading process needs to read out the program's instructions and then write them into the target memory, so a memory that supports write operations is required. PSRAM is still under testing at present; considering that mem-test is not very large, we can try loading mem-test into the 8KB SRAM mentioned above.

Who performs this loading operation? We certainly need to complete the loading before mem-test executes, but we also want to maintain the feature of "fetching instructions from flash after the NPC resets", so that the simulation environment does not interfere too much and the loading operation can also be performed on a real chip. Therefore, we have to use the gap between the NPC reset and the actual execution of mem-test to perform the loading, and it is not hard to think of it — that is the bootloader! In other words, we need to extend the bootloader's functionality to load all the code and data of the program into SRAM, and then jump to SRAM for execution.

Completely test PSRAM access

Extend the bootloader's functionality to fully load mem-test into SRAM, and then execute mem-test. Some hints are as follows:

1. You also need to load the read-only data segment into SRAM together; it may contain some data needed during code execution, such as jump tables
2. If you find that mem-test's code is too large to fit in SRAM, you can try using the compilation option -Os to instruct gcc to optimize for code size
3. The implementation of code loading also needs to handle several details that you can already understand at present; if you ignore them, then learn from debugging, since real projects in the future will be just like this

Afterwards, let mem-test test all 4MB of storage space in the PSRAM; you will find that the execution efficiency of mem-test has improved considerably, and the test can be completed in about 10 minutes.

Although SRAM access is fast, its capacity is small and cannot hold most programs. After completely testing PSRAM access, we can also consider letting the bootloader fully load the program into PSRAM, thereby improving the execution efficiency of the program.

Load the program into PSRAM through the bootloader for execution

You have already loaded mem-test into SRAM through the bootloader, so loading a program into PSRAM is not difficult. However, to make full use of SRAM, we can allocate the stack in SRAM to improve the efficiency of function calls and accesses to local variables.

Try executing microbench at test scale; you will find that compared with executing in flash, loading it into PSRAM and executing it yields a considerable performance improvement.

Execute RT-Thread on PSRAM

The capacity of PSRAM is now sufficient to run RT-Thread. Try loading RT-Thread into PSRAM through the bootloader for execution.

If the program is large, the bootloader's process of loading the program will also take longer, because the bootloader itself executes in flash. Likewise, can we load the "bootloader loads the program" code into a memory with higher access efficiency first, and then execute the "bootloader loads the program" functionality? This is in fact a multi-stage loading process of the bootloader. For ease of distinction, we can split the entire bootloader's work into two parts: FSBL (first stage bootloader) and SSBL (second stage bootloader). When the system is powered on, FSBL, SSBL, and the program to be run are all located in flash; the first to execute is FSBL, which is responsible for loading SSBL from flash into other memory and then jumping to SSBL; then SSBL is responsible for loading the program to be run next from flash into PSRAM, and then jumping to the program and executing it. In fact, SSBL's code is not large, so we can have FSBL load SSBL into SRAM for execution, making SSBL execute faster.

Implement the two-stage loading process of the bootloader

Implement FSBL and SSBL according to the above functionality. To know the range of SSBL in flash, you may need to place SSBL in a separate section; for details, you can refer to the relevant code in start.S.

In addition, because the bootloader splits the loading work into multiple stages, you also need to consider whether the target object is accessible in the current stage.

Internal structure of SDRAM chips

If we want to further improve the efficiency of accessing DRAM chips, we need to consider changing the serial bus between the controller and the chip into a parallel bus. For example, the internal structure of the DRAM chip model MT48LC16M16A2 is shown in the figure below. This chip has 39 pins, including:

• CLK, CKE - clock signal and clock enable signal
• CS#, WE#, CAS#, #RAS - command signals
• BA[1:0] - bank address
• A[12:0] - address
• DQ[15:0] - data
• DQM[1:0] - data mask, named DQML and DQMH in the figure below

Unlike the frequency-divided SCK in the SPI protocol, the clock signal CLK here is usually directly driven by the clock of the DRAM controller. This type of DRAM is called synchronous DRAM, i.e., SDRAM (Synchronous Dynamic Random Access Memory). SDRAM chips have become the mainstream memory chips today; the earliest commercially available SDRAM chip was released in 1992, belonging to SDR SDRAM (Single Data Rate SDRAM), which transfers data once per clock. The chip model MT48LC16M16A2 shown in the figure above belongs to the SDR SDRAM chips. Starting from 1997, DDR SDRAM (Double Data Rate SDRAM) appeared, which can transfer data on both the rising and falling edges of the clock, thereby increasing the data transfer bandwidth; afterwards, DDR2, DDR3, DDR4, and DDR5 appeared in succession, each further increasing the data transfer bandwidth through different technologies. In contrast to synchronous DRAM, there is also asynchronous DRAM, whose bus signals do not include a clock. At present, asynchronous DRAM has basically been replaced by SDRAM, so when we discuss DRAM today, it almost always refers to SDRAM.

Unlike the SPI bus interface of the PSRAM chip, the pins of a traditional DRAM chip include information such as the bank address, which requires the DRAM controller to understand the internal organization of the storage array in the DRAM chip in order to know what information to drive onto the address-related pins.

The storage array of a DRAM chip is a multi-dimensional structure, which logically consists of several matrices, and one matrix is also called a memory bank. A matrix element in a memory bank consists of several storage cells, and each storage cell contains one transistor and one capacitor, used to store 1 bit of information. A matrix element in a memory bank needs to be specified jointly by a row address and a column address. For example, the above DRAM chip has 4 memory banks, each with 8192 rows and 512 columns, and a matrix element in a memory bank has 16 storage cells, so the capacity of this DRAM chip is 4 * 8192 * 512 * 16 = 256Mb = 32MB.

When reading, first select a target memory bank according to the bank number. Then activate a row in the target memory bank through the row address: the sense amplifier in the target memory bank detects the charges in all the storage cells of this row, thereby learning whether each storage cell stores 1 or 0. A sense amplifier is mainly composed of a pair of cross-coupled inverters, so it can store information; hence, the sense amplifier in a memory bank is also called a row buffer, which can store a row of information. Afterwards, according to the column address, data is selected from the row buffer of the target memory bank and output to the outside of the DRAM chip chip as the read result. When writing, first write the data to be written into the corresponding positions of the row buffer, and then charge or discharge the capacitors in the corresponding storage cells, thereby transferring the contents of the row buffer into the storage cells. If another row of data is to be accessed, before activating the other row, the information of the currently activated row must first be written back to the storage cells; this process is called precharge.

Physical implementation of DRAM chips

Considering the limitations of physical implementation, overly long traces introduce large delays. Therefore, from the perspective of physical implementation, a memory bank of a DRAM chip is further divided into multiple subarrays. However, the structures and access methods of these subarrays are transparent to the outside of the chip, and the DRAM controller does not need to care about them when sending commands to the DRAM chip. Interested students can read this article to further understand the physical structure inside a DRAM chip.

After understanding the internal organization of DRAM chips, we can sort out the commands of DRAM chips. The commands of different versions of SDRAM differ slightly; the following table lists the commands of SDR SDRAM:

CS#	RAS#	CAS#	WE#	Command	Meaning
1	X	X	X	COMMAND INHIBIT	No command
0	1	1	1	NO OPERATION	NOP
0	0	1	1	ACTIVE	Activate a row in the target memory bank
0	1	0	1	READ	Read a column from the target memory bank
0	1	0	0	WRITE	Write a column in the target memory bank
0	1	1	0	BURST TERMINATE	Stop the current burst transfer
0	0	1	0	PRECHARGE	Close the activated row in the memory bank (precharge)
0	0	0	1	AUTO REFRESH	Refresh
0	0	0	0	LOAD MODE REGISTER	Set the Mode register

The above commands involve the concept of "burst transfer", which refers to a transaction containing multiple consecutive data transfers, where one data transfer is called a "beat". Taking reading as an example, from the DRAM chip receiving the READ command to reading out the data from the storage array and transferring it onto the DQ bus, several cycles are generally needed, and this delay is called CAS latency (some textbooks translate it as the CAS latency period). Taking the MT48LC16M16A2 model above as an example, depending on the working frequency, the CAS latency can be 1 to 3 cycles, i.e., from receiving the READ command to reading out 16 bits of data onto the DQ bus, there is a latency of 1 to 3 cycles. Assuming a CAS latency of 2 cycles, if burst transfer is not used, reading out 8 bytes requires sending the READ command 4 times, totaling 12 cycles; if burst transfer is used, only 1 READ command needs to be sent, costing 6 cycles.

// normal
  1   1   1   1   1   1   1   1   1   1   1   1
|---|---|---|---|---|---|---|---|---|---|---|---|
  ^       |   ^       |   ^       |   ^       |
  |       v   |       v   |       v   |       v
 READ   data READ   data READ   data READ   data

// burst
  1   1   1   1   1   1
|---|---|---|---|---|---|
  ^       |   |   |   |
  |       v   v   v   v
 READ    1st 2nd 3rd 4th

The WRITE command also supports burst transfer, i.e., to write 8 bytes, the data to be written can be transmitted continuously in the 3 cycles immediately following the issue of the WRITE command. As for how many beats a burst transfer transaction contains, it can be set through the Mode register. The Mode register can also set parameters such as the CAS latency.

ysyxSoC integrates an implementation of the SDR SDRAM controller (referred to as the SDRAM controller below) and maps the SDRAM storage space to the CPU's address space 0xa000_0000~0xbfff_ffff. The code of the SDRAM controller is located in the ysyxSoC/perip/sdram/core_sdram_axi4/ directory, and it uses the AXI4 bus protocol. To facilitate early testing, we have encapsulated its core part ysyxSoC/perip/sdram/core_sdram_axi4/sdram_axi4_core.v into the APB bus protocol (see ysyxSoC/perip/sdram/sdram_top_apb.v) and connected it into the APB Xbar of ysyxSoC. The SDRAM controller translates the received bus transactions into commands sent to the SDRAM chip. We have chosen to simulate the SDRAM chip model MT48LC16M16A2. ysyxSoC has already connected the SDRAM chip to the SDRAM controller, but does not provide code for the SDRAM chip. To use the SDRAM in ysyxSoC, you still need to implement a behavioral model of the SDRAM chip.

Implement the simulation behavioral model of the SDRAM chip

You need to implement the simulation behavioral model of the MT48LC16M16A2 chip. Specifically, you need to implement the commands that the SDRAM controller will send, among which the PRECHARGE and AUTO REFRESH commands are related to the electrical characteristics of the storage cells and need not be considered in the simulation environment, so they can be implemented as NOP. In addition, the Mode register only needs to implement CAS Latency and Burst Length; other fields can be ignored.

Specifically, if you choose Verilog, you need to implement the corresponding code in ysyxSoC/perip/sdram/sdram.v; if you choose Chisel, you need to implement the corresponding code in the sdramChisel module of ysyxSoC/src/device/SDRAM.scala, and modify Module(new sdram) in ysyxSoC/src/SoC.scala to instantiate the sdramChisel module.

For other details, please RTFM by referring to the relevant manual, or RTFSC by referring to the code of the SDRAM controller. To correctly implement the communication between the SDRAM controller and the SDRAM chip, you do not need to modify the code of the SDRAM controller.

After implementation, test a small segment of SDRAM access (such as 4KB) with mem-test to check whether your implementation is correct.

Completely test SDRAM access

Before further loading programs into the SDRAM for execution, we still first test accesses to all the storage space of the above SDRAM chip with mem-test. Completing this test takes about 1 hour.

Load a program into SDRAM for execution

Let the bootloader load the program into the SDRAM and execute it. After that, try executing microbench and RT-Thread on the SDRAM.

Extension of SDRAM chips

As mentioned above, the capacity of the MT48LC16M16A2 SDR SDRAM chip is 32MB. But today's memory modules are often as large as 4GB; how is this achieved? Even though today's mainstream memory modules use more advanced processes and the capacity of a single memory chip has increased, 4GB is still not achieved by a single memory chip alone. In fact, this is accomplished by combining and extending multiple memory chips in certain dimensions. If you have observed the structure of a memory module, you will find that there are multiple memory chips on one memory module, and a memory module is a special PCB board: it integrates multiple memory chips with a standard size specification and uses a DIMM interface, so that it can be inserted into the memory slots of the motherboard.

We know that, without considering the physical organization, memory is a two-dimensional matrix, where each row stores one word and the row number is the address. From this perspective, the extension of multiple chips is nothing more than extension in two dimensions: one dimension is to store more bits of information at one address, called bit extension; the other dimension is to increase the range of addresses, called word extension.

The idea of bit extension is that different chips simultaneously read data from the same address, which not only increases the memory capacity but also improves the memory access bandwidth. If the word length of a chip (i.e., the bit width of the DQ signal) is smaller than the bus data bit width, bit extension can significantly improve the efficiency of data transfer. For example, for a 64-bit CPU, the bus data bit width is usually no less than 64 bits; using 4 MT48LC16M16A2 chips with a word length of 16 bits, 64 bits can be read out after one CAS latency, which is even more efficient than burst transfer:

  1   1   1
|---|---|---|
  ^       |
  |       v
 READ   data[15:0]  (chip0)

  1   1   1
|---|---|---|
  ^       |
  |       v
 READ   data[31:16] (chip1)

  1   1   1
|---|---|---|
  ^       |
  |       v
 READ   data[47:32] (chip2)

  1   1   1
|---|---|---|
  ^       |
  |       v
 READ   data[63:48] (chip3)

Another example is the DDR4 SDRAM memory module model MTA9ASF51272PZ. The word length of the chips on the module is 8 bits, but through bit extension with 8 chips, 64 bits can be read and written at a time.

Extend the data bit width of the SDRAM controller to 32 bits

Instantiate the submodule of 2 SDRAM chips to simulate the scenario of bit extension of 2 SDRAM chips. To do this, you need to modify the following:

• The bit width of some signals in the SDRAM bus interface
  • If you use Chisel, you can directly modify the definition of SDRAMIO
  • If you use Verilog
    • If you do not plan to participate in tape-out, you can directly modify the corresponding signal bit widths in ysyxSoC/build/ysyxSoCFull.v
    • If you plan to participate in tape-out, you cannot directly modify ysyxSoC/build/ysyxSoCFull.v, otherwise the automated tape-out test flow will overwrite your modifications to ysyxSoC/build/ysyxSoCFull.v; instead, you can add some commands to the ysyxSoC/Makefile, which can automatically modify the signal bit widths after generating ysyxSoC/build/ysyxSoCFull.v
• The internal implementation of the SDRAM controller

After implementing bit extension, there is no need to access the SDRAM chips through burst transfer mode; after one CAS latency, 32 bits of data can be read out from the extended chips. Try running some benchmarks to compare the performance changes before and after bit extension.

However, these performance improvements do not come for free. Bit extension requires the number of DQ pins to grow linearly, which requires careful calculation for chips that care about cost. In addition, even if cost is not a concern, the performance improvement brought by bit extension is also limited by other factors in the system, such as the bus data bit width: if one transfer on the bus can be at most 64 bits, then even if the word length of the memory chips is extended to 512 bits, no significant overall benefit will be gained, just as the flow of a water pipe is limited by its narrowest segment.

The other dimension is word extension, whose idea is to distribute different addresses to different chips, thereby increasing the memory access capacity. For example, we can use 2 MT48LC16M16A2 chips of 32MB to form a memory of 64MB capacity through word extension. The above word extension only needs to add 1 bit of address on the memory bus, and the growth in the number of pins has a logarithmic relationship with the storage capacity, so the overhead is not large compared with bit extension. But intuitively, word extension cannot directly improve the memory access bandwidth; we will continue to discuss this problem in the following sections.

Word extension for the SDRAM controller

Instantiate a submodule of 4 SDRAM chips in total, where bit extension is performed between two pairs of SDRAM chips, and then word extension is performed on the results of the bit extension. You also need to modify the SDRAM bus interface and the internal implementation of the controller.

After implementation, test all the extended storage space with mem-test. Completing this test takes several hours.

Note that your modifications may cause the SDRAM controller to fail to meet the electrical characteristics of the SDRAM chips, preventing it from running on real SDRAM chips. However, testing the electrical characteristics of SDRAM requires a more complex simulation environment. To keep things simple, we do not require the modified SDRAM controller to run correctly on a real board.

Access more peripherals

All the work above has been carried out around TRM. Finally, let's see how to support IOE.

GPIO

First, let's add GPIO support. GPIO is arguably the simplest peripheral; its essence is a wire connecting the inside and the outside of the chip, and obviously it needs to occupy the pins of the chip. Through GPIO, the chip can directly output internal signals to the outside of the chip, to drive some simple devices, such as the LEDs on the board; the chip can also obtain some simple external states through GPIO, such as the states of the DIP switches and buttons on the board.

Obviously, the software running on the CPU cannot directly access a pin of the chip; a GPIO controller is also needed to provide the abstraction of device registers for the GPIO functionality. However, for a GPIO controller, the functions of these device registers are very simple: it only needs to use registers in the circuit to store the states of the corresponding pins. Specifically, for output pins, their states are directly driven by a certain bit stored in the register; for input pins, they determine the state of a certain bit in the register.

ysyxSoC integrates a GPIO controller with an APB bus interface and maps it to the CPU's address space 0x1000_2000~0x1000_200f. We have only allocated a 16-byte address space to the GPIO controller, supporting up to 128 pins, which is usually sufficient. To see the effect of GPIO, we will reconnect the NVBoard project you encountered in the pre-learning stage. Considering the peripherals provided by NVBoard, those suitable for GPIO include 16 LEDs, 16 DIP switches, and 8 7-segment displays. Therefore, we allocate the register space of the GPIO controller as follows:

Address	Usage
0x0	16-bit data, driving the 16 LEDs respectively
0x4	16-bit data, obtaining the states of the 16 DIP switches respectively
0x8	32-bit data, of which every 4 bits drive one 7-segment display
0xc	Reserved

However, ysyxSoC does not provide the specific internal implementation of the GPIO controller; we leave it to you as an assignment.

Update NVBoard

We updated NVBoard to version 1.0 at 2024/01/11 01:00:00, which not only added UART functionality, but also greatly improved processing performance. Some of the subsequent lab content will require you to use the NVBoard functionality. If you obtained the NVBoard code before the above time, you can get the new version with the following command:

cd nvboard
git pull origin master

To run the new version of NVBoard, you may need to clear some old compilation results first.

Implement the flowing LED effect on NVBoard through a program

You need to do the following:

1. Implement the register used to drive the LEDs in the GPIO controller. Specifically, if you choose Verilog, you need to implement the corresponding code in ysyxSoC/perip/gpio/gpio_top_apb.v; if you choose Chisel, you need to implement the corresponding code in the gpioChisel module of ysyxSoC/src/device/GPIO.scala, and modify Module(new gpio_top_apb) in ysyxSoC/src/GPIO.scala to instantiate the gpioChisel module
2. Connect to NVBoard, binding the GPIO output pins in the top-level module ysyxSoCFull to the LEDs
3. Write a test program that writes data to the above register at intervals, thereby achieving the flowing LED effect

Unlike the pre-learning stage, the flowing LEDs here are no longer directly controlled by the hardware circuit, but by software, and you already understand all the details involved.

Read the DIP switch states through a program

Similar to the LEDs, let the program read out the states of the DIP switches. You can set a 16-bit binary password in the program; when the program starts, it continuously queries the states of the DIP switches, and only when the states of the DIP switches match the above password does the program continue to execute.

Display the student ID on the 7-segment displays through a program

Read the student ID from the student ID CSR, convert it into 8 hexadecimal digits, and use them to drive the 8 7-segment displays respectively.

UART

We have previously tested the serial port output functionality with the help of the UART16550 controller, but at that time the transmitting end of the serial port only output through the $write system task in the UART16550 controller code, without involving the process of encoding characters and serially transmitting them through wires to the receiving end. NVBoard integrates a serial port terminal, and with NVBoard, we can now experience this process!

The serial port terminal in NVBoard is very simple; it only supports the 8N1 serial transmission configuration. As for the baud rate, since NVBoard has no concept of clock frequency, it is described by a divisor, i.e., how many cycles a bit needs to be maintained during data transmission. The divisor in NVBoard does not support runtime configuration, but it can be adjusted by modifying the code. You can modify it in the UART constructor in nvboard/src/uart.cpp, in two specific ways:

1. Modify the initial value of the divisor member
2. Call the set_divisor() function to set it

Connect the TX pin of the serial port to NVBoard

You only need to modify the NVBoard constraint file to bind the TX pin of the serial port to the serial port terminal of NVBoard. For how to bind, you can refer to the examples provided by NVBoard. Since the serial port controller has already been integrated into ysyxSoC, you do not need to modify the RTL code.

After successfully binding the pin, configure the divisor of the UART in NVBoard according to the actual situation. Note that the divisor of the UART in NVBoard and the divisor in the divisor register of the UART16550 are not exactly the same; there is a certain relationship between them, and you need to sort it out through RTFSC or RTFM.

Then, rerun any test program with serial port output. You will see that the serial port output not only appears in the command line terminal, but also in the serial port terminal in the upper right corner of NVBoard.

NVBoard also supports the input functionality of the serial port. After connecting to NVBoard, you can test the serial port input functionality that was previously inconvenient to test. Binding the pin is not difficult, but we also need to consider how the upper-layer software uses it. Specifically, you also need to add the functionality of the abstract register UART_RX to the IOE of riscv32e-ysyxsoc: read a character from the serial port device; if there is no character, return 0xff.

Bug fix

We fixed an issue related to implementation-defined behavior in the key test of am-tests. If you obtained the code of am-kernels before 2024/01/11 00:00:00, please obtain the new version of the code:

cd am-kernels
git pull origin master

Test the input functionality of the serial port through NVBoard

After binding the RX pin of the serial port and adding the above abstract register in the IOE, run the key test in am-tests to test whether it can obtain the key information through the RX port of the UART.

For how to input through the RX port of the UART in NVBoard, you can refer to the examples provided by NVBoard. In addition, you may need to implement some functionality in the IOE; you can RTFSC for details.

After adding the UART RX related functionality to the IOE, we can try typing commands into RT-Thread through the serial port.

Bug fix

We fixed the bug in rt-thread-am where it got stuck when entering invalid commands, and also made msh obtain keys through polling. If you obtained the code of rt-thread-am before 2024/01/11 01:50:00, please obtain the new version of the code:

cd rt-thread-am
git pull origin master

After obtaining the new version of the code, you also need to regenerate some configuration files:

cd rt-thread-am/bsp/abstract-machine
rm rtconfig.h
make init

Then recompile and run.

Type commands into RT-Thread through the serial port

For this purpose, we need to let RT-Thread call the IOE functionality just implemented. Modify the serial port input functionality in the BSP so that, after reading the built-in string, it obtains characters from the UART RX through the IOE.

PS/2 keyboard

You have already done the keyboard-related digital circuit experiments in the pre-learning stage. Now let's integrate the previous experiment content into ysyxSoC. ysyxSoC integrates a PS2 keyboard controller with an APB bus interface and maps it to the CPU's address space 0x1001_1000~0x1001_1007. However, ysyxSoC does not provide the specific internal implementation of the PS2 keyboard controller, and you need to implement it. We allocate the register space of the PS2 controller as follows:

Address	Usage
0x0	8-bit data, reading the keyboard scan code; if there is no key information, reads 0
Others	Reserved

Let riscv32e-ysyxsoc read keyboard keys from NVBoard

You need to do the following:

1. Implement the PS2 keyboard controller. If you choose Verilog, you need to implement the corresponding code in ysyxSoC/perip/ps2/ps2_top_apb.v; if you choose Chisel, you need to implement the corresponding code in the ps2Chisel module of ysyxSoC/src/device/Keyboard.scala, and modify Module(new ps2_top_apb) in ysyxSoC/src/Keyboard.scala to instantiate the ps2Chisel module
   • Compared with having the keyboard controller translate the scan codes into other encodings, we recommend that the software obtain the scan codes and perform the translation: this not only reduces the complexity of the hardware design, but also improves flexibility
2. Connect to NVBoard and bind the relevant pins
3. Add code in the AM IOE to read the key information from the PS2 keyboard controller and translate it into the keyboard codes defined by AM
   • For the keyboard scan codes, you can refer to the relevant information on the digital circuit experiments
   • Note that the scan codes of some keys contain extension codes, such as PAGEUP, and you need to recognize them correctly

After implementation, run the key test in am-tests to check whether your implementation is correct.

VGA

You should have seen the VGA display effect in NVBoard during the pre-learning stage. Now let the program use ysyxSoC to output pixel information to NVBoard. ysyxSoC integrates a VGA controller with an APB bus interface and maps it to the CPU's address space 0x2100_0000~0x211f_ffff. This address space is actually a frame buffer; writing pixel information into it will output it to the VGA area of NVBoard. The VGA screen resolution provided by NVBoard is 640x480. However, ysyxSoC does not provide the specific internal implementation of the VGA controller, and you need to implement it.

Review how VGA works

If you did not come across VGA-related content in the digital circuit experiments during the pre-learning stage, we suggest you first complete the relevant experiment content to understand the working principle of VGA; otherwise, you may encounter difficulties when designing the VGA controller.

Let riscv32e-ysyxsoc output pixel information to NVBoard

You need to do the following:

1. Implement the VGA controller, which continuously outputs the contents of the frame buffer to the screen through the physical VGA interface. If you choose Verilog, you need to implement the corresponding code in ysyxSoC/perip/vga/vga_top_apb.v; if you choose Chisel, you need to implement the corresponding code in the vgaChisel module of ysyxSoC/src/device/VGA.scala, and modify Module(new vga_top_apb) in ysyxSoC/src/VGA.scala to instantiate the vgaChisel module
   • Regarding the frame buffer, for now you can temporarily implement it in a simple way like SRAM. But note that in real situations, this implementation is costly: taking the 640x480 resolution mentioned above as an example, if each pixel occupies 4 bytes, 1.17MB of SRAM would be needed, which would occupy a considerable amount of tape-out area
2. Connect to NVBoard and bind the relevant pins
3. Add code in the AM IOE to write pixel information into the frame buffer of the VGA controller
   • Since the VGA mechanism provided by NVBoard is automatically refreshed, there is no need to implement the picture synchronization functionality in AM

After implementation, run the picture test in am-tests to check whether your implementation is correct.

A more practical frame buffer implementation

Usually, a frame buffer is allocated in memory; by configuring some registers in the VGA controller, the VGA controller can read pixel information from the memory.

Think about it: if the frame buffer were allocated in memory in the current ysyxSoC configuration, what problems might it cause?

Display games through NVBoard

Try running the typing game and the NES games on NVBoard. Of course, this should be very laggy; our subsequent work is to optimize the performance of the system at the microarchitecture level.

Run AM programs on RT-Thread

After connecting the above devices, we can run other AM programs on RT-Thread, thereby forming a complete SoC computer system!

• The top layer is applications, such as the NES games, which run in RT-Thread in the form of threads
• RT-Thread provides the management of several resources, including physical memory, threads, etc.
  • RT-Thread can also manage many resources, such as files, etc., which are not used by our computer system for now
• The AM runtime environment provides the functional abstractions of TRM, IOE, and CTE, to support RT-Thread running on bare metal
• The RISC-V instruction set provides concrete instructions, the MMIO mechanism, and the exception handling mechanism, to implement the concrete functionalities of AM's TRM, IOE, and CTE
• NPC implements the functionality of the RISC-V instruction set
• ysyxSoC integrates the NPC, so that the NPC communicates with the various device controllers through the bus in the SoC
• NVBoard simulates the functionality of the development board, provides the physical implementations of the devices, and interacts with the device controllers through pins

Update RT-Thread

We added the functionality of integrating other AM programs to RT-Thread at 2024/01/18 20:00:00. If you obtained the code of rt-thread-am before the above time, please obtain the new version of the code:

cd rt-thread-am
git pull origin master

Run other AM programs through RT-Thread

Referring to the optional task "Running AM programs on RT-Thread" in Phase 1 of PA4, try starting other AM programs such as the NES games through RT-Thread in riscv32e-ysyxsoc.

ChipLink - Inter-chip bus protocol

All the device controllers above are located in the same SoC, so they occupy a certain amount of tape-out area. If a device controller is complex (such as a modern DDR controller), it will cost a lot of tape-out budget. In fact, we can extend the bus outside the chip: two chips can communicate with each other while following the same bus protocol. In this way, the chip we design can access the devices on other finished chips, which on the one hand does not occupy our tape-out area, thereby saving tape-out cost, and on the other hand also reduces the complexity of verification and the risk of tape-out, after all, the functionalities on the finished chips have already been quite thoroughly verified.

If the peer chip is an FPGA, we can also gain flexible expansion capability: we only need to burn the device controllers into the FPGA, and the chip can access these devices through the inter-chip bus protocol. Even if there are bugs in the device controllers, no catastrophic consequences will result; you only need to re-burn the FPGA after fixing the bugs.

However, these interfaces occupy the pins of the chip. Considering an AXI bus with both address bit width and data bit width of 32 bits, the signals araddr, awaddr, rdata, and wdata alone occupy 128 pins; adding the various control signals, the total is about 150 pins; if the data bit width is 64 bits, more than 200 pins would be needed in total. Therefore, if we directly connect the AXI bus to the outside of the chip, we would most likely need to adopt a more expensive packaging solution, which goes against the original intention of saving cost.

If we want to send AXI requests outside the chip through fewer pins, we can only time-multiplex the pins: by decomposing the signals of an AXI request and transmitting a part of them each time, a complete AXI request is transmitted over multiple cycles. For example, if only 32 pins are used, we can agree to transmit the 32-bit write address at time T0, the 32-bit write data at time T1, and the other control signals at time T2. If the peer chip also follows the same convention, it can, according to the convention, reassemble the information received on the 32 pins over these 3 cycles into an AXI write request, thereby achieving the effect of transmitting an AXI write request to another chip over 3 cycles.

Such a convention is in fact a set of inter-chip bus protocols, which specifies the various details of transmitting AXI requests outside the chip through time-division multiplexing. For ease of description, we call the inter-chip bus protocol the outer protocol, and the bus protocol that is decomposed and transmitted the inner protocol. For example, in the above scenario, AXI is the inner protocol during inter-chip transmission. Of course, the inner protocol does not have to be AXI; requests of other protocols can also be decomposed and transmitted.

For example, the inter-chip bus protocol ChipLink can transmit the TileLink bus protocol as its inner protocol across chips. Similar to AXI, TileLink is also full-duplex, i.e., the sender and the receiver can transmit information on the channel at the same time. Therefore, ChipLink is also designed to be full-duplex; in a single direction, in addition to the 32 data signals, there are also clock, reset, and valid signals, and the standard ChipLink protocol needs to occupy 70 pins, much fewer than the pins occupied by transmitting the inner protocol TileLink outside the chip alone.

In fact, the number of pins occupied by ChipLink can be further reduced by decreasing the data signal bit width. For example, when the data bit width is reduced to 8 bits, the ChipLink protocol only needs to occupy 22 pins. However, this comes at the cost of transmission bandwidth: transmitting one inner protocol request takes more cycles, so the amount of effective data transmitted per unit of time also decreases accordingly. If the usage scenario does not require much bandwidth, this approach can be used to save chip packaging cost.

ChipLink provides an open source implementation, but its inner protocol only supports TileLink. However, we can use the adapter bridges in the rocket-chip project to first convert AXI requests into TileLink requests, then transmit the TileLink requests to the peer chip through the ChipLink protocol; after the peer chip reassembles the TileLink requests according to the ChipLink protocol, it converts the TileLink requests back into AXI requests through the adapter bridges, thereby achieving inter-chip transmission of AXI requests.

ysyxSoC integrates the above open source implementation of ChipLink and simulates the scenario of connecting to a peer FPGA chip through ChipLink. The simulated peer FPGA chip contains a 1GB memory, and ysyxSoC maps this space to the CPU's address space 0xc000_0000~0xffff_ffff. In actual use, what devices are contained in this address space is programmable, i.e., we can make full use of the programmability of the FPGA: by updating the bitstream file of the FPGA, different devices can be connected into this address space.

Access the resources of the peer chip through ChipLink

ChipLink is not enabled by default in ysyxSoC, so you need to change the hasChipLink variable to true in the Config object of ysyxSoC/src/Top.scala, regenerate ysySoCFull.v, and simulate.

However, the ChipLink code requires that the reset signal of the simulation top level be maintained for at least 10 cycles; you need to check whether your simulation code meets this condition.

After enabling ChipLink, test the above address space with mem-test. Since the focus of the test is to check whether the NPC can access the peer resources through ChipLink, we do not need to test the entire address space; it is sufficient to test the access of a small segment of the storage space (such as 4KB).

Since the implementation of ChipLink is complex, adding ChipLink will generate a lot of Verilog code, which significantly reduces the simulation efficiency. Therefore, the subsequent lab content does not require you to work with ChipLink enabled. After passing the current test, you can disable ChipLink.

yyz