Note this blog post is a an evolving work in progress. All comments welcome by email to Jonathan.Cameron@huawei.com

The CXL 2.0 specification is a fairly complex beast and as such, software enablement relies on having a suitable platform well in advance of actual hardware. This helps with several aspects: * Specification prove out. * Early enablement to make sure the Linux distributions have support in place when hardware does arrive.

The approach taken to this, spear-headed by the team at Intel is to emulate CXL 2.0 in QEMU and develop the Linux kernel, firmware and tooling support against that.

Enabling similar for arm64 built directly on their work and the required code additions were fairly minimal. However, bringing up a suitable test environment is less trivial and requires collating information from many sources. The purpose of this blog is to bring all that information together in one location.

The key parts needed are: * QEMU * EDK2 * Kernel * Suitable OS Image

Lets assume we have a ~/src/images directory for where we are going to put all the things needed to boot our emulated machine.

Cross compiler

Often available as part of a Linux distribution. I'm going to assume that anyone interested in this blog has one of these on hand already.

EDK2

Tianocore or EDK2 is an open source BIOS. A recent version is needed to enable the pxb-pcie / pxb-cxl PCI expansion bridges used for CXL emulation. Linaro provides a useful set of scripts to aid building suitable firmware images. For this document I used commit f297b7f200107

mkdir tianocore cd tianocore git clone https://git.linaro.org/uefi/uefi-tools.git git clone https://github.com/tianocore/edk2.git git submodule init \ MdeModulePkg/Library/BrotliCustomDecompressLib/brotli git submodule init \ BaseTools/Source/C/BortiCompress/brotli git submodule init \ CryptoPkg/Library/OpensslLib/openssl git submodule update uefi-tools/edk2-build.sh armvirtqemu64 cp Build/ArmVirtQemu-AARCH64/RELEASEGCC5/FV/QEMUEFI.fd \ ~/src/images

Filesystem / OS Image

So normally I'd advocate taking the time to install your favorite Linux distribution on they system you are going to emulate, but for this blog post, let's take a different approach. Abusing cloud images.

There are nice sophisticated things that can be done with cloud-init and CDROM images etc, but it turns out Debian also ships images that skip the cloud-init part and have a root user with no password. Just what we need to trying something like this.

http://cdimage.debian.org/cdimage/cloud/sid/daily/20210504-627/debian-sid-nocloud-arm64-daily-20210504-627.qcow2 which works equally well with less complexity. (I might do a follow up blog on the CDROM method as it was a fun detour and may be applicable to more distributions)

Kernel

For first run, we'll use the mainline kernel.

git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git cd linux make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- defconfig

To avoid the need for messing around with initrd images which is a big waste of time when developing, we'll tweak the config in order to ensure everything we 'need' is built in.

Not all of these are technically necessary but they will probably be helpful if you are developing CXL stuff.

CONFIGACPIHMAT=y CONFIGACPIAPEIPCIEAER=y CONFIGACPIHOTPLUGMEMORY=y CONFIGMEMORYHOTPLUG=y CONFIGMEMORYHOTPLUGDEFAULTONLINE=y CONFIGMEMORYHOTREMOVE=y CONFIGCXLBUS=y CONFIGCXLMEM=y

Build the kernel and copy it to our images directory

make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- cp arch/arm64/boot/Image ~/images

QEMU

The CXL emulation in QEMU is not yet upstream. So get it from Ben Widawsky's gitlab. He's been kind enough to include almost all the patches we need to support CXL on arm64.

git clone https://gitlab.com/bwidawsk/qemu cd qemu

Now make a small change that I haven't shared upstream yet. Note there has been some discussion on the right way to fix this but no firm conclusions yet. https://lore.kernel.org/qemu-devel/20210513124737.00002b2d@Huawei.com/

Fix unaligned undersized access

Signed-off-by: Jonathan Cameron Jonathan.Cameron@huawei.com > —- softmmu/memory.c | 10 ++++++++— 1 file changed, 8 insertions(+), 2 deletions(–)

diff —git a/softmmu/memory.c b/softmmu/memory.c index d4493ef9e4..11c1d54077 100644 —– a/softmmu/memory.c +++ b/softmmu/memory.c @@ -531,6 +531,7 @@ static MemTxResult accesswithadjustedsize(hwaddr addr, unsigned accesssize; unsigned i; MemTxResult r = MEMTXOK; + int offset = 0; if (!accesssizemin) { accesssizemin = 1; @@ -541,7 +542,12 @@ static MemTxResult accesswithadjustedsize(hwaddr addr, /* FIXME: support unaligned access? */ accesssize = MAX(MIN(size, accesssizemax), accesssizemin); – accessmask = MAKE64BITMASK(0, accesssize * 8);
+ if (accesssize > size) { + offset = (addr % accesssize) * 8; + accessmask = MAKE64BITMASK(offset, size * 8); + } else { + accessmask = MAKE64BITMASK(0, accesssize * 8); + }
if (memoryregionbigendian(mr)) { for (i = 0; i < size; i += accesssize) { r |= accessfn(mr, addr + i, value, accesssize, @@ -549,7 +555,7 @@ static MemTxResult accesswithadjustedsize(hwaddr addr,
} else { for (i = 0; i < size; i += accesssize) { – r |= accessfn(mr, addr + i, value, accesssize, i * 8, + r |= accessfn(mr, addr – (addr % accesssize) + i, value, + accesssize, -offset + i * 8, accessmask, attrs); } }

Build QEMU

mkdir -p bin/native cd bin/native ../../configure —target-list=aarch64-softmmu —disable-docs make

Booting the result

cd ~/src/images/ ./qemu-system-aarch64 -M virt -m 4g,maxmem=8G,slots=8 -cpu max \ -smp 4 -kernel Image -nographic -no-reboot \ -bios QEMUEFI.fd \ -drive if=none,file=cxl.qcow2,format=qcow2,id=hd \ -device virtio-blk-pci,drive=hd \ -append 'root=/dev/vda1' \ -object memory-backend-ram,size=4G,id=mem0 \ -numa node,nodeid=0,cpus=0-3,memdev=mem0 \ -object memory-backend-file,id=cxl-mem1,share,mem-path=/tmp/cxltest.raw,size=2G,align=2G \ -device pxb-cxl,busnr=128,id=cxlbus1,uid=1,len-window-base=1,window-base[0]=0x4c0000000,memdev[0]=cxl-mem1 \ -device cxl-rp,bus=cxlbus1,id=cxlrp,chassis=0,slot=1 \ -device cxl-type3,bus=cxlrp,memdev=cxl-mem1,lsa=cxl-mem1,id=cxl-mem0,size=2G

Login is [username: root, password:]

You will probably want to be able to scp to the board later, so take this opportunity to generate host keys

dpkg-reconfigure openssh-server Also, make your image nice an insecure by editing /etc/ssh/sshd_config

To set

PasswordAuthentication yes PermitEmptyPassword yes

Data Object Exchange support

QEMU

Need the patches from Avery Design. https://lore.kernel.org/qemu-devel/1619454964-10190-1-git-send-email-cbrowy@avery-design.com/

If using the b4 tool then

b4 am -t 1619454964-10190-1-git-send-email-cbrowy@avery-design.com

Will fetch the patches. Note the final test data appears to be corrupt, so given we don't need it use

git am —skip

Kernel

Need the Data Object Exchange patches from https://lore.kernel.org/linux-pci/20210524133938.2815206-1-Jonathan.Cameron@huawei.com/T/#t

b4 am -t 20210524133938.2815206-1-Jonathan.Cameron@huawei.com

which apply to https://git.kernel.org/pub/scm/linux/kernel/git/cxl/cxl.git/commit/?h=next&id=35c32e3095d396c750f5cdfdaa94cba83d9b23c6

These both enable CDAT access via /sys/bus/cxl/devices/mem0/CDAT and a IOCTL interface (which will probably never be merged) for which there is a doetest program in tools/pci/

Cross compile the doetest program and test the DOE with:

$ ./doetest -f /dev/pcidoe/doe[0000:81:00.0]_190 -l VID: 0x1, Protocol 0 VID: 0x1e98, Protocol 0x2