Compiling Nanos for RISC-V

[Yifferpi]

I'm trying to compile nanos for the RISC-V architecture and there does not seem to be any sort of documentation on it yet. I've tried

make PLATFORM=riscv-virt

and I get the following error message:

..nanos/src/riscv64/interrupt.c: Assembler messages:
nanos/src/riscv64/interrupt.c:99: Error: illegal operands `mv a0,fp'

I know riscv is not officially/fully supported yet. Also according to #1651 some instructions are not supported yet (not mv though).

I would be happy if you could maybe provide some instructions on how to compile for/to riscv, whatever is working at this point. If it isn't working yet, maybe some pointers to what is yet missing and what would need to be done to compile a working image would be helpful.

Thanks

[sanderssj]

Hi @Yifferpi,
What platform and version are you building on? You need a fairly recent toolchain to build RISC-V nanos, and your error message suggests yours might be too old. You can try building your own toolchain (see https://github.com/riscv-collab/riscv-gnu-toolchain) or use a more recent Linux distribution which does have a new enough toolchain already packaged. There is also a write-up with some basic instructions on building and running nanos on RISC-V at https://nanovms.com/dev/tutorials/nanos-and-riscv.

[Yifferpi]

I have an Ubuntu bionic beaver which is indeed a bit older

Linux yiffXPS13 4.15.0-169-generic #177-Ubuntu SMP Thu Feb 3 10:50:38 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

and the following versions

$ riscv64-linux-gnu-gcc --version
riscv64-linux-gnu-gcc (Ubuntu 8.4.0-1ubuntu1~18.04) 8.4.0
$ riscv64-linux-gnu-as --version
GNU assembler (GNU Binutils for Ubuntu) 2.30

I will try to build my own toolchain then.

[Yifferpi]

So building my own toolchain helped with regard to the illegal operand error.

There are now different errors though which I assume are missing dependencies.. The errors are the following:

riscv64-unknown-linux-gnu-ld: /home/yiff/gitrepos/eth/kernelcomp/nanos/output/platform/riscv-virt/platform/riscv-virt/service.o: in function `vm_exit':
/home/yiff/gitrepos/eth/kernelcomp/nanos/platform/riscv-virt/service.c:105: undefined reference to `memcpy

and

CC	/home/yiff/gitrepos/eth/kernelcomp/nanos/output/test/runtime/test/runtime/hw.o
In file included from /home/yiff/gitrepos/eth/kernelcomp/nanos/test/runtime/hw.c:1:
/home/yiff/gitrepos/eth/kernelcomp/riscv64-target-root/usr/include/stdio.h:27:10: fatal error: bits/libc-header-start.h: No such file or directory
   27 | #include <bits/libc-header-start.h>
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.

The directory /home/yiff/gitrepos/eth/kernelcomp/riscv64-target-root is where I extracted the file riscv64-target-root.tar.gz to
and which I set the env variable NANOS_TARGET_ROOT to.

$ tree -L 1 /home/yiff/gitrepos/eth/kernelcomp/riscv64-target-root
/home/yiff/gitrepos/eth/kernelcomp/riscv64-target-root
├── bin -> usr/bin
├── boot
├── dev
├── etc
├── home
├── lib -> usr/lib
├── media
├── mnt
├── opt
├── proc
├── root
├── run
├── sbin -> usr/sbin
├── srv
├── sys
├── tmp
├── usr
└── var

From what I understand, this means that the target root fs that I downloaded from https://storage.googleapis.com/testmisc/riscv64-target-root.tar.gz (like described in the linked Tutorial) does not contain all necessary files. How can I add them? (some sort of chroot to install them in the directory or use a different root fs overall?) Or is there a newer version specifically for cross-compiling nanos?

[sanderssj]

I tried building and running nanos on bionic and did not have a problem aside from the manually built riscv toolchain using a slightly different prefix than the one specified in the makefile. Did you set CC manually by chance to fix the toolchain name? Some of the nanos build objects (such as certain tests) are actually compiled and run on the host architecture rather than the target platform, and I think that might be the source of the errors as they do not use the cross-compiler and must have the host's normal CC. If that's the case, remove the output directory containing the built artifacts, make sure the repo is back to the default state and try running the build like this:

CROSS_COMPILE=riscv64-unknown-linux-gnu- PLATFORM=riscv-virt make

Let me know if that fixes your issue or if you still get those errors. Thanks!

[Yifferpi]

It worked.
I previously forgot to mention: I did noticed the difference in the cross compiler prefix and adjusted it (though directly in the Makefile).

Properly deleting the output directory and resetting the repo helped the the errors above. Compilation still did not terminate without error. In the final step (MKFS and image tag), it still failed due to some sort of missing dependency libpthread.so.1 in the toolchain.

What seems to have made the difference in the end was recompiling the toolchain: according to their instruction, the directory /opt/riscv/bin must be added to the path before running ./configure --prefix=/opt/riscv which I did not do the first time around.

Thank you very much for the concise help.

Maybe as a follow up: is there any integration with ops yet? As I understand, the outputs from the nanos compilation should also work with ops by copying them to ~/.ops or by declaring them in the config.json?

[Yifferpi]

Actually, after playing around with it for some more, there is still a problem persisting:

The following compilation error that I mentioned earlier keeps coming up:

CC	/home/yiff/gitrepos/eth/kernelcomp/nanos/output/test/runtime/test/runtime/hw.o
In file included from /home/yiff/gitrepos/eth/kernelcomp/nanos/test/runtime/hw.c:1:
/home/yiff/gitrepos/eth/kernelcomp/riscv64-target-root/usr/include/stdio.h:27:10: fatal error: bits/libc-header-start.h: No such file or directory
   27 | #include <bits/libc-header-start.h>
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.

I did not realize earlier but it occurs whenever I change the TARGET to hw rather than webg, i.e., I use

CROSS_COMPILE=riscv64-unknown-linux-gnu- PLATFORM=riscv-virt TARGET=hw make  -j 8

or set TARGET=hw in the Makefile

Like mentioned earlier, I wonder whether this is error means there is some missing dependency in the riscv64-target-root root fs? How do I fix it? Use a different target root?

Also, what would be the target for an output suitable for ops?

[sanderssj]

Hi @Yifferpi,
I'm glad you've been able to make progress. I was able to reproduce your issue, and it is a flag issue rather than a problem with the target root. As I've looked into it I've found some peculiar differences in compilation and behavior between the riscv toolchain from github and the ones packaged into newer versions of Ubuntu. Below is a patch that you can use that should get you past your current set of errors, at least. I have found there are still some remaining issues around VDSO and the time related syscalls, however. They only appear when building with the github toolchain, but you may also run into them. I am currently investigating that problem.

In regards to your earlier question, we will soon add support to ops for running RISC-V programs locally, but that will still require the user to build on a RISC-V Linux box or cross-compile their program for RISC-V to create a suitable binary for ops to use.

I will have a fix for these issues soon, and I am also adding a check for the RISC-V gcc version so that the makefile will error if the gcc RISC-V support is not new enough.

diff --git a/test/runtime/Makefile b/test/runtime/Makefile
index b6f9ce42..df0fd068 100644
--- a/test/runtime/Makefile
+++ b/test/runtime/Makefile
@@ -317,8 +317,8 @@ dummy:
 include ../../rules.mk

 ifneq ($(CROSS_COMPILE),)
-CFLAGS+=	--sysroot $(TARGET_ROOT) -I$(TARGET_ROOT)/usr/include
-LDFLAGS+=	--sysroot=$(TARGET_ROOT) -L$(TARGET_ROOT)/usr/lib/$(ARCH)-linux-gnu
+CFLAGS+=	--sysroot $(TARGET_ROOT) -isystem $(TARGET_ROOT)/usr/include -isystem $(TARGET_ROOT)/usr/include/$(ARCH)-linux-gnu
+LDFLAGS+=	--sysroot=$(TARGET_ROOT) -L$(TARGET_ROOT)/usr/lib/$(ARCH)-linux-gnu -B$(TARGET_ROOT)/usr/lib/$(ARCH)-linux-gnu
 ifeq ($(ARCH),riscv64)
 LDFLAGS+=	-Wl,--eh-frame-hdr
 endif

[Yifferpi]

Hi @sanderssj
The patch worked wonders. The hw target is running now. Thanks alot.

I understand that a cross-compiled binary will be needed for ops. I assume the missing part for ops is the boot image? I am looking forward to the ops support, but I'm already in a good place now to try adaptations on the riscv by testing directly with nanos rather than through ops.

Thank you very much for the support.

[francescolavra]

I assume the missing part for ops is the boot image?

Since at the moment the RISC-V port runs only under qemu, we don't really need a full blown boot image, because we can pass the kernel file directly to the qemu command line. So currently the missing part for ops only involves starting qemu with the RISC-V-specific command line.

[sanderssj]

Hi @Yifferpi,
The patch you've been using (as well as other fixes) have made it into the master branch: #1698

[Yifferpi]

I don't know if this should belong into another issue: I wanted to run existing tests and issued the following command:

CROSS_COMPILE=riscv64-unknown-linux-gnu- PLATFORM=riscv-virt make test-noaccel -j 8

It fails with the following syntax error:

[...]
[...]nanos/output/test/unit/bin/buffer_test
[...]nanos/output/test/unit/bin/buffer_test: 1: [...]nanos/output/test/unit/bin/buffer_test: Syntax error: word unexpected (expecting ")")
Makefile:146: recipe for target 'test' failed
make[2]: *** [test] Error 2
make[2]: Leaving directory '[...]nanos/test/unit'
Makefile:7: recipe for target 'test' failed
make[1]: *** [test] Error 2
make[1]: Leaving directory '[...]nanos/test'
Makefile:98: recipe for target 'test-noaccel' failed
make: *** [test-noaccel] Error 2

I tried to add the problematic unittest to the SKIP_TEST variable in the unittest Makefile and the error persists in the next unittest. Normal compilation with various TARGETs work fine.

[francescolavra]

Currently the test and test-noaccel Makefile targets work only when building for the native architecture of the computer where they are run.
I think the Makefiles should be changed so that unit tests (which are plain executable files and are not run under Nanos) are built for the native architecture regardless of the PLATFORM and CROSS_COMPILE variables specified by the user; in any case, you can run the runtime tests (which are run as Nanos unikernel images) for RISC-V with CROSS_COMPILE=riscv64-unknown-linux-gnu- PLATFORM=riscv-virt make runtime-tests-noaccel