Howto test CXL enablement on Arm64 using QEMU
Note: Parts of this blog are outdated / wrong. I might fix it at somepoint soon!
Note this blog post is a an evolving work in progress. All comments welcome by email to Jonathan.Cameron@huawei.com
The CXL 2.0 specification is a fairly complex beast and as such, software enablement relies on having a suitable platform well in advance of actual hardware. This helps with several aspects: * Specification prove out. * Early enablement to make sure the Linux distributions have support in place when hardware does arrive.
The approach taken to this, spear-headed by the team at Intel is to emulate CXL 2.0 in QEMU and develop the Linux kernel, firmware and tooling support against that.
Enabling similar for arm64 built directly on their work and the required code additions were fairly minimal. However, bringing up a suitable test environment is less trivial and requires collating information from many sources. The purpose of this blog is to bring all that information together in one location.
The key parts needed are: * QEMU * EDK2 * Kernel * Suitable OS Image
Lets assume we have a ~/src/images directory for where we are going to put all the things needed to boot our emulated machine.
Cross compiler
Often available as part of a Linux distribution. I'm going to assume that anyone interested in this blog has one of these on hand already.
EDK2
Tianocore or EDK2 is an open source BIOS. A recent version is needed to enable the pxb-pcie / pxb-cxl PCI expansion bridges used for CXL emulation. Linaro provides a useful set of scripts to aid building suitable firmware images. For this document I used commit f297b7f200107
mkdir tianocore cd tianocore git clone https://git.linaro.org/uefi/uefi-tools.git git clone https://github.com/tianocore/edk2.git git submodule init \ MdeModulePkg/Library/BrotliCustomDecompressLib/brotli git submodule init \ BaseTools/Source/C/BortiCompress/brotli git submodule init \ CryptoPkg/Library/OpensslLib/openssl git submodule update uefi-tools/edk2-build.sh armvirtqemu64 cp Build/ArmVirtQemu-AARCH64/RELEASEGCC5/FV/QEMUEFI.fd \ ~/src/images
Filesystem / OS Image
So normally I'd advocate taking the time to install your favorite Linux distribution on they system you are going to emulate, but for this blog post, let's take a different approach. Abusing cloud images.
There are nice sophisticated things that can be done with cloud-init and CDROM images etc, but it turns out Debian also ships images that skip the cloud-init part and have a root user with no password. Just what we need to trying something like this.
http://cdimage.debian.org/cdimage/cloud/sid/daily/20210504-627/debian-sid-nocloud-arm64-daily-20210504-627.qcow2 which works equally well with less complexity. (I might do a follow up blog on the CDROM method as it was a fun detour and may be applicable to more distributions)
Kernel
For first run, we'll use the mainline kernel.
git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git cd linux make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- defconfig
To avoid the need for messing around with initrd images which is a big waste of time when developing, we'll tweak the config in order to ensure everything we 'need' is built in.
Not all of these are technically necessary but they will probably be helpful if you are developing CXL stuff.
CONFIGACPIHMAT=y CONFIGACPIAPEIPCIEAER=y CONFIGACPIHOTPLUGMEMORY=y CONFIGMEMORYHOTPLUG=y CONFIGMEMORYHOTPLUGDEFAULTONLINE=y CONFIGMEMORYHOTREMOVE=y CONFIGCXLBUS=y CONFIGCXLMEM=y
Build the kernel and copy it to our images directory
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- cp arch/arm64/boot/Image ~/images
QEMU
The CXL emulation in QEMU is not yet upstream. So get it from Ben Widawsky's gitlab. He's been kind enough to include almost all the patches we need to support CXL on arm64.
git clone https://gitlab.com/bwidawsk/qemu cd qemu
Now make a small change that I haven't shared upstream yet. Note there has been some discussion on the right way to fix this but no firm conclusions yet. https://lore.kernel.org/qemu-devel/20210513124737.00002b2d@Huawei.com/
Fix unaligned undersized access
Signed-off-by: Jonathan Cameron Jonathan.Cameron@huawei.com > —- softmmu/memory.c | 10 ++++++++— 1 file changed, 8 insertions(+), 2 deletions(–)
diff —git a/softmmu/memory.c b/softmmu/memory.c index d4493ef9e4..11c1d54077 100644 —– a/softmmu/memory.c +++ b/softmmu/memory.c @@ -531,6 +531,7 @@ static MemTxResult accesswithadjustedsize(hwaddr addr, unsigned accesssize; unsigned i; MemTxResult r = MEMTXOK; + int offset = 0; if (!accesssizemin) { accesssizemin = 1; @@ -541,7 +542,12 @@ static MemTxResult accesswithadjustedsize(hwaddr addr, /* FIXME: support unaligned access? */ accesssize = MAX(MIN(size, accesssizemax), accesssizemin); – accessmask = MAKE64BITMASK(0, accesssize * 8);
+ if (accesssize > size) { + offset = (addr % accesssize) * 8; + accessmask = MAKE64BITMASK(offset, size * 8); + } else { + accessmask = MAKE64BITMASK(0, accesssize * 8); + }
if (memoryregionbigendian(mr)) { for (i = 0; i < size; i += accesssize) { r |= accessfn(mr, addr + i, value, accesssize, @@ -549,7 +555,7 @@ static MemTxResult accesswithadjustedsize(hwaddr addr,
} else { for (i = 0; i < size; i += accesssize) { – r |= accessfn(mr, addr + i, value, accesssize, i * 8, + r |= accessfn(mr, addr – (addr % accesssize) + i, value, + accesssize, -offset + i * 8, accessmask, attrs); } }
Build QEMU
mkdir -p bin/native cd bin/native ../../configure —target-list=aarch64-softmmu —disable-docs make
Booting the result
cd ~/src/images/ ./qemu-system-aarch64 -M virt -m 4g,maxmem=8G,slots=8 -cpu max \ -smp 4 -kernel Image -nographic -no-reboot \ -bios QEMUEFI.fd \ -drive if=none,file=cxl.qcow2,format=qcow2,id=hd \ -device virtio-blk-pci,drive=hd \ -append 'root=/dev/vda1' \ -object memory-backend-ram,size=4G,id=mem0 \ -numa node,nodeid=0,cpus=0-3,memdev=mem0 \ -object memory-backend-file,id=cxl-mem1,share,mem-path=/tmp/cxltest.raw,size=2G,align=2G \ -device pxb-cxl,busnr=128,id=cxlbus1,uid=1,len-window-base=1,window-base[0]=0x4c0000000,memdev[0]=cxl-mem1 \ -device cxl-rp,bus=cxlbus1,id=cxlrp,chassis=0,slot=1 \ -device cxl-type3,bus=cxlrp,memdev=cxl-mem1,lsa=cxl-mem1,id=cxl-mem0,size=2G
Login is [username: root, password:]
You will probably want to be able to scp to the board later, so take this opportunity to generate host keys
dpkg-reconfigure openssh-server Also, make your image nice an insecure by editing /etc/ssh/sshd_config
To set
PasswordAuthentication yes PermitEmptyPassword yes
Data Object Exchange support
QEMU
Need the patches from Avery Design. https://lore.kernel.org/qemu-devel/1619454964-10190-1-git-send-email-cbrowy@avery-design.com/
If using the b4 tool then
b4 am -t 1619454964-10190-1-git-send-email-cbrowy@avery-design.com
Will fetch the patches. Note the final test data appears to be corrupt, so given we don't need it use
git am —skip
Kernel
Need the Data Object Exchange patches from https://lore.kernel.org/linux-pci/20210524133938.2815206-1-Jonathan.Cameron@huawei.com/T/#t
b4 am -t 20210524133938.2815206-1-Jonathan.Cameron@huawei.com
which apply to https://git.kernel.org/pub/scm/linux/kernel/git/cxl/cxl.git/commit/?h=next&id=35c32e3095d396c750f5cdfdaa94cba83d9b23c6
These both enable CDAT access via /sys/bus/cxl/devices/mem0/CDAT and a IOCTL interface (which will probably never be merged) for which there is a doetest program in tools/pci/
Cross compile the doetest program and test the DOE with:
$ ./doetest -f /dev/pcidoe/doe[0000:81:00.0]_190 -l VID: 0x1, Protocol 0 VID: 0x1e98, Protocol 0x2