Survey

Please take the Kata Containers survey:

• https://openinfrafoundation.formstack.com/forms/kata_containers_user_survey

This will help the Kata Containers community understand:

• how you use Kata Containers
• what features and improvements you would like to see in Kata Containers

Libseccomp Notices

The kata-agent binaries inside the Kata Containers images provided with this release are
statically linked with the following GNU LGPL-2.1 licensed libseccomp library.

• libseccomp

The kata-agent uses the libseccomp v2.5.5 which is not modified from the upstream version.
However, in order to comply with the LGPL-2.1 (§6(a)), we attach the complete source code for the library.

Kata Containers builder images

• agent (on all its different flavours): quay.io/kata-containers/builders:agent-65c32735e-8724d7dee-x86_64
• Kernel (on all its different flavours): quay.io/kata-containers/builders:kernel-4fc34323a-x86_64
• OVMF (on all its different flavours): quay.io/kata-containers/builders:ovmf-2ee03b5dc-x86_64
• QEMU (on all its different flavurs): quay.io/kata-containers/builders:qemu-fe5adae5d-x86_64
• shim-v2: quay.io/kata-containers/builders:shim-v2-go-1.19.3-rust-1.72.0-04d021bd1-x86_64
• tools: quay.io/kata-containers/builders:tools-77540503f-d915a79e2-9e01732f7-x86_64
• virtiofsd: quay.io/kata-containers/builders:virtiofsd-1.72.0-musl-2205fb9d0-x86_64

Installation

Follow the Kata installation instructions.

What's Changed

• docs: Update links in the Documentation Requirements document by @GabyCT in #9307
• gha: Update journal log names for kubernetes artifacts by @GabyCT in #9309
• gha: Fix nydus namespace clean up by @GabyCT in #9265
• Dragonballl: introduce MTRR regs support by @studychao in #9311
• tests: static checker: Add announce message by @jodh-intel in #9259
• agent: Add guest-pull to the list of agent features in announce() by @ChengyuZhu6 in #9312
• docs: Update libseccomp instructions in Developers Guide by @GabyCT in #9324
• Revert "release: Skip --generate-notes for this release" by @fidencio in #9321
• runtime-rs: ch: Implement full thread/tid/pid handling by @dborquez in #9255
• versions: Update nydus-snapshotter to v0.13.11 by @fidencio in #9337
• runtime-rs: Enable qemu on s390x by @BbolroC in #9280
• agent: Refactor unit tests to leverage rstest for parameterization by @ChengyuZhu6 in #9313
• runtime-rs/dragonball: add support building kernel with upcall and GPU hotplug by @Apokleos in #9244
• agent:image: Refactor code to improve memory efficiency of image service by @ChengyuZhu6 in #9325
• scripts: Fix unbound variables in k8s setup script by @GabyCT in #9329
• workflows: Build agent-opa for more archs by @stevenhorsman in #9356
• Remove additional links to tests directory by @cmaf in #9346
• docs: Add documents for kata guest image management by @ChengyuZhu6 in #9341
• Only tag and publish the release when it is fully ready by @gkurz in #9326
• Support to set timeout to pull large image in guest by @ChengyuZhu6 in #9332
• k8s: confidential: Update cpuid to its latest release by @fidencio in #9349
• runtime: remove unimplemented CoCo configurations by @fitzthum in #8046
• genpolicy: reduce policy debug prints by @danmihai1 in #9347
• runtime: remove stream copy infinite loop by @danmihai1 in #9367
• agent: Fix errors in make check by @c3d in #9345
• gha: Update journal log names for nerdctl artifacts by @GabyCT in #9358
• kata-agent: Change order of guest hook and bind mount processing by @Apokleos in #9275
• kata-agent: enabling cgroups-v2 by systemd.unified_cgroup_hierarchy by @Apokleos in #9383
• versions: Remove runc version information by @GabyCT in #9365
• gha: add GENPOLICY_PULL_METHOD by @Redent0r in #9385
• docs: Remove stale kernel information by @GabyCT in #9344
• versions: Remove conmon information from versions.yaml by @GabyCT in #9397
• gha: Define GH_PR_NUMBER variable in gha run k8s common script by @GabyCT in #9409
• tests: k8s-job: wait for job successful create by @danmihai1 in #9411
• gha: ensure unique resource group name by @Redent0r in #9413
• bugfix and refactor device increate count by @Apokleos in #8782
• tdx: Update TDX artefacts to be used with the Ubuntu 23.10 / CentOS 9 stream OSVs. by @fidencio in #8840
• tests: Support for kbs setup on kcli by @ldoktor in #9273
• metrics: Improve latency test cleanup by @GabyCT in #9419
• GHA: Implement secondary GITHUB_WORKSPACE cleanup on 1st failure by @BbolroC in #9415
• qemu: show the thread name when enable the hypervisor.debug option by @deagon in #9402
• docs: kata-manager: Update with latest details by @jodh-intel in #9372
• port attestation agent from CCv0 branch to main branch by @LindaYu17 in #8870
• agent:image: Support different pause image in the guest for guest pull by @ChengyuZhu6 in #9369
• gha: Bump various actions to use Node.js 20 by @gkurz in #9421
• katautils: check number of cores on the system intead of go runtime by @egernst in #9331
• tests: k8s: improve the Agent Policy tests by @danmihai1 in #9398
• docs: adding an initial CI documentation by @beraldoleal in #8988
• genpolicy: Add optional toggle to pull images using containerd by @Redent0r in #9185
• add onednn and openvino ml-benchmarks by @dborquez in #9391
• gha: Fix indentation in gha run script by @GabyCT in #9450
• tests: Improve the kbs_k8s_delete function by @GabyCT in #9423
• tests: k8s: inject agent policy failures by @danmihai1 in #9439
• agent: Fix the issue with the "test_new_fs_manager" test by @justxuewei in #9457
• CC: run guest-pull tests on non-TEE jobs by @wainersm in #9424
• gha: Define unbound PULL TYPE variable by @GabyCT in #9454
• agent: shutdown vm on exit when agent is used as init process by @alex-matei in #9430
• CI: Enable GHA cri-containerd workflow for runtime-rs with QEMU by @BbolroC in #9403
• kernel: Adjust s390x config for confidential containers by @BbolroC in #9469
• ci.ocp: Increase the MCP update time by @ldoktor in #9404
• version: Add coco name and version for {image,initrd} for s390x by @BbolroC in #9471
• gha: make run-kata-coco-tests inherit secrets by @wainersm in #9479
• runtime-rs: refactor qemu driver by @pmores in #9353
• tests: k8s: inject agent policy failures (part2) by @danmihai1 in https://github.com/kata-co...

Survey

Please take the Kata Containers survey:

• https://openinfrafoundation.formstack.com/forms/kata_containers_user_survey

This will help the Kata Containers community understand:

• how you use Kata Containers
• what features and improvements you would like to see in Kata Containers

Libseccomp Notices

The kata-agent binaries inside the Kata Containers images provided with this release are
statically linked with the following GNU LGPL-2.1 licensed libseccomp library.

• libseccomp

The kata-agent uses the libseccomp v2.5.5 which is not modified from the upstream version.
However, in order to comply with the LGPL-2.1 (§6(a)), we attach the complete source code for the library.

Kata Containers builder images

• agent (on all its different flavours): quay.io/kata-containers/builders:agent-65c32735e-8724d7dee-x86_64
• Kernel (on all its different flavours): quay.io/kata-containers/builders:kernel-4fc34323a-x86_64
• OVMF (on all its different flavours): quay.io/kata-containers/builders:ovmf-6bb2ea819-x86_64
• QEMU (on all its different flavurs): quay.io/kata-containers/builders:qemu-0538bbfc4-x86_64
• shim-v2: quay.io/kata-containers/builders:shim-v2-go-1.19.3-rust-1.72.0-a13eecf7f-x86_64
• tools: quay.io/kata-containers/builders:tools-b3b00e00a-9ef59488d-5bad18f9c-x86_64
• virtiofsd: quay.io/kata-containers/builders:virtiofsd-1.72.0-musl-2205fb9d0-x86_64

Installation

Follow the Kata installation instructions.

What's Changed

• metrics: Add parallel udp iperf3 benchmark by @GabyCT in #8278
• runtime-rs: fix a typo in device manager by @ZizhengBian in #8294
• AArch64: runtime: use pcie root port to do pci/pcie device hotplug by @jongwu in #7647
• dragonball: add metrics support for balloon device by @lisongqian in #7697
• kata-manager: Add clh config to containerd config file by @amshinde in #8281
• gha: add dependencies for spell checker by @cmaf in #8317
• runtime-rs: Add default configuration file for cloud-hypervisor by @amshinde in #8250
• tests/git-helper: cancel any previous rebase left halfway by @wainersm in #8322
• agent: use open_tree()/move_mount() to set up bind mounts between containers directly. by @h56983577 in #8033
• dragonball: add metrics support for legacy device by @lisongqian in #7695
• kata-runtime/kata-ctl: Add security details to output by @jodh-intel in #8314
• dragonball: add tracing feature for dragonball by @lisongqian in #7831
• utils: kata manager: Fix version checks by @jodh-intel in #8323
• Enable fio checkmetrics by @dborquez in #8202
• network: Fix network attach for ipvlan and macvlan by @amshinde in #8334
• agent: Skip flaky create_tmpfs on s390x by @BbolroC in #8289
• runtime-rs: Log system enhancement by @TimePrinciple in #8311
• docs: Fix broken links by @cmaf in #8255
• cargo: Agent cargo.lock updated by @amshinde in #8351
• release: Fully migrate from hub to gh by @gkurz in #8308
• gha: Add workflow to close stale PRs by @fidencio in #8348
• kata-manager: Fix deployment of containerd on architectures other than amd64. by @brianwang12 in #7057
• Docs: Fix Dragonball link by @sazzy4o in #8285
• gha: stale: Fix typo and allow manually triggering it by @fidencio in #8368
• kata-manager: Accept only "lts" or "active" as containerd versions by @fidencio in #8365
• runtime-rs: update device pci info for vfio and virtio-blk devices by @amshinde in #8284
• Updating containerd to a GogoProtobuf free version by @beraldoleal in #8061
• tests: fixes permission denied when running test by @beraldoleal in #8217
• runtime-rs: ch: Simplify VSOCK error handling by @jodh-intel in #8386
• agent: Restrict device access at upper node of container's cgroup by @justxuewei in #7531
• runtime-rs: Update status for pause and resume by @cmaf in #8023
• network: Fix network hotplug for ipvlan and macvlan endpoints for qemu and add tests by @amshinde in #8367
• runtime: Fix TestCheckHostIsVMContainerCapable unstablity issue by @justxuewei in #8389
• Upgrade to Cloud Hypervisor v36.0 by @likebreath in #8379
• gha: Fix regex used to get kubectl version from the k3s version by @fidencio in #8411
• kata-deploy: Allow users to set hypervisor annotations by @fidencio in #8404
• agent: update AGENT_THREADS metrics value by @gaohuatao-1 in #8370
• runtime-rs: fix a typo in shm by @studychao in #8169
• kata-manager: Add support for Docker CLI installation by @fidencio in #8376
• Update release process documentation by @gkurz in #8309
• utils: kata-manager: Ensure only one download URL by @jodh-intel in #8374
• docs: add agent policy documentation by @danmihai1 in #8406
• dragonball: Introduce vhost-net device by @justxuewei in #7675
• runtime-rs: ch: Fix TDX by @jodh-intel in #8419
• metrics: Fix function that completely stops kata containers before running a test by @dborquez in #8338
• utils: kata-manager: Add option to list versions by @jodh-intel in #8383
• ci: Re-add tracing tests and move docker/nerdctl to the basic-ci-amd64.yaml file by @fidencio in #8174
• gha: Remove docker and nerdctl tests from ci.yaml by @justxuewei in #8432
• runtime: Improve vCPU allocation for the VMMs by @fidencio in #7623
• kernel: Fix vsock packets drop when the driver initializes by @alex-matei in #8431
• dragonball: Remove vhost-net dependency on virtio-net by @justxuewei in #8426
• tests|gha: add nightly tests for s390x by @BbolroC in #7987
• gha: Keep kata tarballs for 15 days by @ldoktor in #8460
• tests: Enable stressng scalability test by @GabyCT in #8421
• metrics: Add iperf udp information to README by @GabyCT in #8453
• tests|gha: add containerd and k8s tests for s390x by @BbolroC in #7931
• StratoVirt: add support for a lightweight VMM StratoVirt in Kata by @WenyuanLau in #7796
• Fixes make check errors by @beraldoleal in #8345
• runitme-rs/bugfix: kata pod with multi-containers sharing one direct volume by @Apokleos in #8332
• kata-deploy: Set a default value for ALLOWED_HYPERVISOR_ANNOTATIONS by @BbolroC in #8478
• dragonball: Uniform the spelling of Virtio by @justxuewei in #8465
• Dragonball: add PCI bus and PCI interrupt support in mptable Spec by @studychao in #8451
• CC: Remote hypervisor merge to main by @stevenhorsman in #7046
  *...

This release was mistakenly deleted by @fidencio while working on the new release process.
Thankfully we had the tag, and the release has been re-created based on the tag.

Release 3.2.0

kata-containers Changes

The biggest change in 3.2.0 is the conversion of CI to GitHub actions as in the main development branch. This is part of the initiative to deprecate the test repository and to stop using Jenkins for CI.

Shortlog

224ae84 release: Kata Containers 3.2.0
2cda69b release: Adapt kata-deploy for 3.2.0
305e603 actions: Move all the checkout actions to v4
52a985e release: Always use actions/checkout to ensure we're in a git repo
dc0fe5d actions: release: Use GH cli instead of hub
93c7d16 ci: k8s: Fix bogus firecracker check in k8s-credentials-secrets.bat
12b8cbb tests: Adjust timeout for agent stability test
37c99a4 tests: Enable agent stability test
92f283f runtime: Validate hypervisor section name in config file
8cf5506 metrics: fixes common.sh function to always return true
544f261 metrics: skips docker restart when it is not installed or is masked.
26c6ca9 metrics: removing trailing comma characters from json file.
0e0aabf metrics: removal of reference in the documentation to the dax test.
5d911db tests: Remove unused function from scability test
a380437 tests: Fix path for versions yaml for soak parallel test
4495a79 tests: Enable scability test for stability CI
961daee scripts: Use install_yq from the kata-containers repo
9b48525 release: tag_repos: Stop tagging / updating the tests repo
668c897 runtime: fix reading cgroup stats of sandboxes
11e2f2a versions: Bump virtiofsd to v1.8.0
9eb8723 clh: arm: Use static_sandbox_resource_mgmt=true
e7579d2 runtime/qemu: Rework QMP/HMP support
f0278f4 runtime/virtiofsd: Drop all references to "--cache=none"
4679aa7 runtime/qemu: Pass "--xattr" to virtiofsd instead of "-o xattr"
03d712a runtime: Allow virtio_fs_extra_args annotation
e051309 runtime/vc: runPrestartHooks should ignore GetHypervisorPid failure
c17cbd3 runtime: fail early when starting docker container with FC
7e6f801 runtime: run prestart hooks before starting VM for FC
fa824af qemu: tdx: Workaround SMP issue with TDX 1.5
07471cd qemu: tdx: Adapt to the TDX 1.5 stack
2f28866 versions: tdx: Update Kernel to 6.2 + TDX
a36064c versions: tdx: Update TDVF to the "edk2-stable202302"
65e0b99 versions: tdx: Update QEMU to v7.2 + TDX v1.10
9ce8ee6 runtime/fc: fix image/initrd annotation handling
f86bfe0 runtime/clh: fix image/initrd annotation handling
59fae42 runtime/qemu: fix image/initrd annotation handling
ef65c57 kata-agent: use default filemode for block device when it is set to 0
93609aa deps: Bump dependent crate versions
7ff98da gha: Add install dependencies for stability tests
ef49db5 gha: Add general dependencies to stability tests
a818f62 tests: Add soak parallel stability test
602c56c tests: Enable soak parallel test
a195539 ci: k8s: set KUBERNETES default value
c4456c2 tests: run k8s-volume on a given node
58ad833 tests: run k8s-file-volume on a given node
a54bdd0 tests: exec_host() now gets the node name
0eaf81c tests: add get_one_kata_node() to tests_common.sh
5f2c7c7 ci: k8s: set KATA_HYPERVISOR default value
7fceb21 ci: k8s: configurable deploy kata timeout
c4b0f1f ci: k8s: shellcheck fixes to gha-run.sh
6fb40ad kata-deploy: re-format kata-[deploy|cleanup].yaml
5cd2e94 ci: k8s: run_tests() for kcli
56cebfb ci: k8s: add deploy-kata-kcli() to gh-run.sh
6b76d21 ci: k8s: add cleanup-kcli() to gha-run.sh
308ce26 ci: k8s: set default image for deploy_kata()
c3b91ed ci: k8s: create k8s clusters with kcli
33791f0 metrics: stops kata components and k8s deployment when test finishes
621e6e6 gha: combine coco jobs into a single yaml
fe52c09 gha: combine basic amd64 jobs into a single yaml
301a7d9 gha: ci: Revert tracing test PR to unbreak CI
c1da29b ci: Port runk tests to this repo
63be808 ci: Add placeholder for runk tests
6541969 ci: Move tracing tests here
5d232c8 ci: Add placeholder for tracing tests
619ef16 ci: Create a function to install docker
16e31dd metrics: Use jq tool to pretty-print json metrics output
1f9a4e9 metrics: Enables FIO test for kata containers
fe4f72e gha: Add containerd stability tests to ci yaml
7963298 gha: Add stability gha run script
a4e0929 gha: Add stability tests workflow for gha
be3a3c2 gha: arm64: Ensure the builder is arm64-builder
f20164d packaging: tools: Remove set -x leftover
1941d87 packaging: release: Mention newly added images
95da1c7 packaging: tools: Fix container image env var name
508016f packaging: Allow passing the TOOLS_CONTAINER_BUILDER
bb1efe0 packaging: stable-3.2: Remove everything related to agent policy
892c9f2 gha: Build the kata-agent as part of our workflows
a586b8c packaging: Build the kata-agent
766a5fa agent: Allow specifying DESTDIR and AGENT_POLICY via env vars
050a426 packaging: Add get_agent_image_name()
3770b20 gha: Fix k0s deployment
cf254bc tests: Add general stability fixes
1edf2d9 tests: Add agent stability test
a8eec39 tests: Add cassandra stress in stability tests
240c584 tests: Add stressng dockerfile for stability tests
e95d3b1 tests: Add stressor CPU test for stability tests
4393f55 metrics: Add stability test for kata CI
362adea metrics: Fix general check static warnings
16c349e docs: Update url in kata vra document
5800be5 ci: Build src/tools components as part of our tests / releases
41b509e kata-deploy: Build components from src/tools
a5d7ba6 static-build: Add scripts to build content from src/tools
d503daf packaging: Add get_tools_image_name()
b2e432c packaging: Use git abbreviated hash
c22fdb4 metrics: Increase qemu jitter value
8a1af86 metrics: Increase jitter value for clh
f3fcf6c metrics: Add checkmetrics for latency test
ce03e9f metrics: Add qemu latency value limit
cd82a35 metrics: Add latency value limits for kata CI
1709f99 ci: kata-monitor: Move tests over
a50c7f1 ci: Add placeholder for kata-monitor tests
c42d196 ci: Make install_kata aware of container engines
5017435 ci: Create a generic install_crio function
98e9434 ci: Add install_cni_plugins helper
c61b488 ci: Modify containerd default config
7c4617c metrics: Add init_env function to latency test
e106ecd metrics: Fix latency yamls path
665805c metrics: Fix spelling warnings
b0c9b42 metrics: Fix metrics README
c28a0a0 metrics: Fix C-Ray documentation
48a9b4a ci: crio: Trail '\r' from exec_host() output
2de1c8b ci: crio: Enable default capabilities
d1d3c7c kata-deploy: Fix CRI-O detection
0de3216 kata-deploy: Add k0s support
468a321 ci: crio: Pass -y to apt
3f2780f metrics: Add latency benchmark for gha
73a084a metrics: Enable latency test in gha run script
cf3abd3 local-build: Fix .docker ownership before build-payload
8b607ff gha: Add pandoc as a dependency for static checks
6a9384e gha: Install hunspell for static checks
a11e886 ci: Trigger payload-after-push on workflow_dispatch
390bde3 ci: Actually enable the CRI-O tests
f2953e6 ci: k8s: rke2: Use sudo to call systemd
08bdb6b ci: k8s: Add a CRI-O test
b41fa6d ci: k8s: Add a method to install CRI-O
67fef9d ci: k8s: k0s: Allow passing parameters to the k0s installer
2c3f130 ci: kata-deploy: Fix runner name
7a8d848 ci: Enable kata-deploy tests for all the supported k8s flavours
7fc2f7d ci: kata-deploy: Add the ability to deploy rke2
59a4b00 ci: kata-deploy: Add the ability to deploy k0s
1a605c3 ci: kata-deploy: Add deploy-k8s argument to gha-run.sh
19ee6c9 ci: kata-deploy: Expland tests to run on k0s / rke2
03a8bed ci: kata-deploy: Add placeholder for tests on GARM
f09c255 ci: kata-deploy: Export KUBERNETES env var
abe9dc9 ci: Move deploy_k8s() to gha-run-k8s-common.sh
ea64896 ci: Properly set K8S_TEST_UNION
7892e04 ci: Add first letter of the K8S_TEST_HOST_TYPE to resource group name
882d7d7 ci: Create clusters in individual resource groups
b09a3f8 metrics: Add parallel bandwidth limit for qemu
63e8c38 metrics: Enable parallel bandwidth iperf limit
f3c42ff nydus: Temporarily skip tests on dragonball
49c1a37 nydus: Use kata-${KATA_HYPERVISOR} instead of kata
ae55c0b static-build: Fix arch error on nydus build
65e5bfe tests: nydus: Update nydus tests
079ab1e versions: Bump nydus and nydus-snapshotter to its latest release
d9e9107 gha: nydus: Populate run()
33a4427 gha: nydus: Populate install_dependencies()
70c1c7d gha: nydus: Actually install kata when install-kata is called
30efa3e gha: nydus: Get rid of nydus{,-snapshotter} install from nydus_test.sh
9ad6000 tests: nydus: Add timeout to the crictl calls
6d9b8e2 tests: nydus: Add uid / namespace to the nydus container / sandbox
fd5935d tests: nydus: Decorate some calls with sudo
4b58777 tests: nydus: Adapt "source ..." to GHA
82c5319 tests: nydus: Adapt check to "clh" instead "cloud-hypervisor"
4915605 tests: common: Add install_nydus_snapshotter()
8e4180f tests: common: Add install_nydus()
625a05a ci: static-checks: Clean up static-checks job
9784ded ci: static-checks: Run tests depending on KVM
668b7ef ci: static-checks: Move "sudo make test" to the new test matrix
4b660a4 ci: static-checks: Move "make test" to the new test matrix
9e614ce runtime-rs: Ensure static-checks-build is a dep of make test
d5d21f4 kata-ctl: Use loop instead ...

kata-containers Changes

A lot of changes have been done as part of this 3.2.0-alpha4 release, and the highlights are:

• runtime-rs improvements for handling block devices
• GPU / VFIO support improvements
• kata-deploy improvements related to custsomising what's being deployed
• A whole bunch of tests migrated from the tests repo to the kata-containers one

Shortlog

743291c release: Fix upload-versions-yaml
bee1a62 metrics: Fix json result for tensorflow
51cd99c metrics: Round axelnet and resnet results
3b883bf metrics: Fix atoi invalid syntax
f9dec11 checkmetrics: Move checkmetrics to gha-run script
53af71c checkmetrics: Add AlexNet value for qemu
a435d36 checkmetrics: Add Resnet value for qemu
a79a3a8 checkmetrics: Add alexnet value for clh
3c32875 checkmetrics: Add Resnet value for clh
08dfaa9 metrics: General improvements to the tensorflow script
63b8534 metrics: Enable Tensorflow metrics for kata CI
1b111a9 gha: release: stage must be defined for arm64 / s390x yamls
684a6e1 Revert "gha: release: stage must be a string"
8a2c201 docs: Update links for pods and kubelet
91e1e61 k8s: Rely on the USING_NFD environment variable passed by the jobs
7c857d3 gha: release: stage must be a string
7edc717 release: Kata Containers 3.2.0-alpha4
6222bd9 tests: Add k8s-file-volume test
187a72d tests: Add k8s-volume test
0c84270 metrics: Add boot time value for qemu
6520dfe metrics: Update boot time for kata metrics
ff22790 metrics: Update runtime and configuration paths
a5d4e33 metrics: Add compare virtiofsd dax script
5e937fa metrics: Update general FIO tests
b0bea47 metrics: Add makefile to report generator
73c57b9 metrics: Add FIO report files for kata metrics
8353aae ci: k8s: Rework get_nodes_and_pods_info()
6ad5d71 ci: k8s: Do not gather node info before running the tests
5261e3a ci: k8s: Group messages to improve readability
9cc6b5f ci: k8s: Get logs from kata-deploy
9d285c6 ci: k8s: Let kata-deploy take care of the runtimeclasses
87568ed gha: Test split out runtimeclasses are in sync with all-in-one file
39192c6 kata-deploy: Print variables passed to the script
0e157be kata-deploy: Allow runtimeclasses to be created by the daemonset
a274333 kata-deploy: Change default values of DEBUG
69535b8 kata-deploy: runtimeclass: Split out entries
9e17106 kata-runtimeClasses: Alphabetically sort the enrties
c8fcd29 runtime-rs: use device manager to handle virtio-pmem
901c192 runtime-rs: support configure vm_rootfs_driver
5d6199f runtime-rs: use device manager to handle vm rootfs
20f1f62 runtime-rs: change block index to 0
314aec7 agent: fix typo in constant
662f875 metrics: Add general FIO makefile
37641a5 metrics: Add example config for fio jobs
3c1044d metrics: Update FIO paths for k8s runner
6177a0d metrics: Add env files for FIO
a459003 metrics: Add fio exec
ea198fd metrics: Add FIO runner k8s
8f7ef41 metrics: Add FIO vendor code
6293c17 metrics: Add FIO benchmark for metrics tests
3aa6c77 gha: dragonball: Run only on the dragonball labeled machine
c5a87ee tests: gha: Add timeout to cluster creation
6daeb08 tests: k8s: Clean up node debuggers after running
b9f100b agent,libs: Remove unused 'mut' keywords
2c8f834 runtime-rs: remove unneeded 'mut' keywords
4703434 tests: k8s: Allow using custom resource group
350f3f7 tests: Import common.bash in run_kubernetes_tests.sh
d7f04a6 tests: k8s: Leave runtimeclass_workloads/ alone
bdde6aa tests: k8s: Split deployment and testing commands
91a0b3b tests: aks: Simply delete cluster when cleaning up
371a118 agent: exclude symlinks from recursive ownership change
c8ac565 cache: kernel: Harmonize commit with fetching side
81775ab cache: kernel: Fix SEV kernel caching
ff4cfcd runk: Add Docker guide to README
4a5ab38 metrics: General improvements to json.bash script
a56f96b kata-deploy: Allow shim creation based on what's passed to the daemonset
717f775 gha: ci: Add skeleton of vfio job
1fc715b s390x: Add AP Attach/Detach test
545de50 vfio: Fix tests
62aa675 vfio: Added better handling of VFIO Control Devices
dd422cc vfio: Remove obsolete HotplugVFIOonRootBus
114542e s390x: Fixing device.Bus assignment
b7c9867 release: Mention the container images used to build the project
d4eba36 kata-deploy-binaries: kernel_cache: Take module_dir into account
7c4b597 ci: nydus: Fix typo in "source"
6a680e2 gha: ci: Add placeholder for the nydus tests as part of the CI
fb4f7a0 gha: nydus: Add a no-op GHA for nydus
4a207a1 gha: nydus: Bring tests as they are from the tests repo
bbd3c1b Dragonball: migrate dragonball-sandbox crates to Kata
e91f5ed ci: cri-containerd: Fix default typo for testContainerStart()
8b8aef0 ci: cri-containerd: Temporarily disable TestContainerSwap
5676700 ci: cri-containerd: Add namespace / uid to the pods
a847736 ci: cri-containerd: Always use sudo to call crictl
99ba86a ci: cri-containerd: Add /usr/local/go/bin to the PATH
7f3b309 ci: cri-containerd: Add function before each function
fde22d6 ci: cri-containerd: Assume podman is always used
9465a04 ci: cri-containerd: Adapt "source ..." to this repo
df8d144 ci: cri-containerd: Remove CI variable
f90570a ci: cri-containerd: Remove unused runc_runtime_bin
c363703 ci: cri-containerd: Remove KILL_VMM_TEST env var
bc4919f ci: cri-containerd: Always run shim-v2 tests
f9e332c ci: cri-containerd: Stop cloning containerd
cfd662f ci: cri-containerd: Remove ununsed SNAP_CI var
d36c339 ci: cri-containerd: Update copyright
b5be8a4 ci: cri-containerd: Move integration-tests.sh as it was
f2e00c9 ci: cri-containerd: Populate install_dependencies()
8979552 versions: Add "latest" field for cri-tools
1bbcbaf ci: Add clone_cri_container()
f66c68a ci: Add install_cri_tools()
4dd8284 ci: Add install_cri_containerd()
ad47d1b ci: Add download_github_project_tarball()
788c562 ci: Add get_latest_patch_release_from_a_github_project()
6742f3a ci: Use function before each install_go.sh function
5eacecf ci: Adjust paths for install_go.sh
8ed1595 ci: Update copyright for install_go.sh
6123d0d ci: Move install_go.sh as it was
8653be7 ci: Do not take cross-build into consideration for kata-arch.sh
6a76bf9 ci: Fix style / identation if kata-arch.sh
7274385 ci: Add function before each kata-arch.sh function
9f6d489 ci: Update copyright for kata-arch.sh
6f73a72 ci: Move kata-arch.sh as it was
3615d73 ci: Add get_from_kata_deps()
3477949 gha: kubernetes: Avoid declaring repo_root_dir
f3738be tests: Use $HOME/go as fallback for $GOPATH
b87ed27 tests: Move ensure_yq to common.bash
124e390 tests: common: Fix quoting when globbing
db77c9a tests: Make install_kata take care of the links
13715db tests: Do not call install_check_metrics when installing kata
630634c ci: k8s: Group logs to make them easier to read
228b30f ci: k8s: Gather node info during the cleanup
81f9954 ci: k8s: Cleanup cluster before deleting it
38a7b53 packaging/tools: Add kata-debug
309e232 cache: kernel: Consider changes in tools/packaging/kernel
ae6e8d2 kata-deploy: Properly get the path of the versions.yaml file
59fdd69 kata-deploy: Add VERSION and versions.yaml to the final tarball
5dddd7c release: Upload versions.yaml as part of the release
87d99a7 versions: Remove "kernel-experimental"
bad3ac8 metrics: Rename C-Ray to cpu performance tests
556e663 metrics: Add disk link to general metrics README
98c1217 metrics: Add C-Ray README
8e7d992 metrics: Add C-Ray Dockerfile
e2ee769 metrics: Add C-Ray performance test
e64edf4 metrics: Add tensorflow function in gha-run script
67a6fff metrics: Enable tensorflow benchmark on gha
8430068 metrics: Add function to memory inside container script
01450de Revert "metrics: Replace backslashes used to escape double quoted key in jq expr."
6a7a323 versions: Bump virtiofsd to v1.7.0
55e2f09 metrics: stop hypervirsor and shim at init_env stage
fad801d ci: k8s: Adapt "source ..." to the new location of gha-run.sh
2ee2cd3 ci: k8s: Move gha-run.sh to the kubernetes dir
88eaff5 ci: tdx: Adjust KUBECONFIG
c09e268 versions: Downgrade SEV(-SNP) kernel back to v5.19.x
950b89f versions: Update kernel to version v6.1.38
6c91af0 agent: Fix exec hang issues with a backgroud process
f72cb2f agent: Remove shadowed function, add slog-term
07810bf agent: Ignore already mounted dev/fs/pseudo-fs
ac5f535 ci: k8s: Bring TDX tests back
8ccc1e5 metrics: Update machine learning documentation
f50d2b0 gha: ci: cri-containerd: Fix KATA_HYPERVSIOR typo
620b945 metrics: Add Tensorflow Mobilenet documentation
a864d0e tests: Add tensorflow mobilenet dockerfile
788d2a2 tests: Add tensorflow mobilenet performance test
468f017 metrics: Replace backslashes used to escape double quoted key in jq expr.
283f809 runtime-rs: Enhancing Device Manager for network endpoints.
ed23b47 tracing: Add tracing to runtime-rs
150e54d runtime-rs: ignore unconfigured network interfaces
59f4731 metrics: Stop running kata-env before kata is properly installed.
3ae02f9 metrics: use rm -f to remove older continerd config file.
2c8dfde kernel: Update kernel config name
64f013f ci: k8s: Enable debug when running the tests
8f4b1df kata-deploy: Give users the ability to run it on DEBUG mode
6787c63 runtime-rs: add parameter for propagation of (u)mount events
62080f8 kata-sys-util: Fix compilation errors
02d99ca static-checks: Make cargo clippy pass.
9824206 agent: Make the static checks pass for agent
61e4032 kata-ctl: Remove all utility functions to get platform protection
a24dbdc kata-sys-util: Move utilities to get platform protection
dacdf7c kata-ctl: Remove cpu related functions from kata-ctl
f5d1957 kata-sys-util: Move additional functionality to cpu.rs
304b9d9 kata-sys-util: Move CPU info functions
6e5679b tests: Add function before function name in co...

kata-containers Changes

This is the ONLY version of Kata Containers 3.1.x that should be
used in production. Previous versions had an issue with the guest
image that's only been fixed as part of this release.

Shortlog

100e9c4 gha: release: Use a specific release of hub
956368e kata-deploy: Change how we get the Ubuntu k8s key
447f368 kata-deploy: Improve shim backup / restore
46bc1f7 kata-deploy: Use apt-key.gpg from k8s.io
984addf kata-deploy: Do not ship the kata tarball
d39aeff kata-deploy: Ensure node is ready after CRI Engine restart
56de5b6 kata-deploy: fix install failing to chmod runtime-rs/bin/*
9de3cf4 kata-deploy: Switch to using an ubuntu image
3c02758 release: Kata Containers 3.1.3
a43f10b release: Adapt kata-deploy for 3.1.3
993ecec virtiofsd: Convert legacy -o sub-options to their -- replacement
2e9125c virtiofsd: Drop -o no_posix_lock
407727e virtiofsd: Stop using deprecated -f option
6668ddb versions: Use ubuntu as the default distro for the rootfs-image
075a311 runtime: sending SIGKILL to qemu

Compatibility with CRI-O

Kata Containers 3.1.3 is compatible with CRI-O

Compatibility with containerd

Kata Containers 3.1.3 is compatible with contaienrd v1.6.8

OCI Runtime Specification

Kata Containers 3.1.3 support the OCI Runtime Specification v1.0.2

Compatibility with Kubernetes

Kata Containers 3.1.3 is compatible with Kubernetes 1.23.1-00

Libseccomp Notices

The kata-agent binaries inside the Kata Containers images provided with this release are
statically linked with the following GNU LGPL-2.1 licensed libseccomp library.

• libseccomp

The kata-agent uses the libseccomp v2.5.4 which is not modified from the upstream version.
However, in order to comply with the LGPL-2.1 (§6(a)), we attach the complete source code for the library.

If you want to use the kata-agent which is not statically linked with the library, you can build
a custom kata-agent that does not use the library from sources.
For the details, please check the developer guide.

Kata Linux Containers image

Agent version: 3.1.3

Default Image Guest OS:

description: |
Root filesystem disk image used to boot the guest virtual
machine.
url: "https://github.com/kata-containers/kata-containers/tools/osbuilder"
architecture:
aarch64:
name: "ubuntu"
version: "latest"
ppc64le:
name: "ubuntu"
version: "latest"
s390x:
name: "ubuntu"
version: "latest"
x86_64:
name: "ubuntu"
version: "latest"
meta:
image-type: "ubuntu"

Default Initrd Guest OS:

description: |
Root filesystem initrd used to boot the guest virtual
machine.
url: "https://github.com/kata-containers/kata-containers/tools/osbuilder"
architecture:
aarch64:
name: "alpine"
version: "3.15"

Do not use Alpine on ppc64le & s390x, the agent cannot use musl because

there is no such Rust target

ppc64le:
name: "ubuntu"
version: "20.04"
s390x:
name: "ubuntu"
version: "20.04"
x86_64:
name: "alpine"
version: "3.15"

Kata Linux Containers Kernel

Kata Containers 3.1.3 suggest to use the Linux kernel v5.19.2
See the kernel suggested Guest Kernel patches
See the kernel suggested Guest Kernel config

Installation

Follow the Kata installation instructions.

Issues & limitations

More information Limitations

kata-containers Changes

In this release we're posting the shortlog between 3.2.0-alpha0 and 3.2.0-alpha3,
as the -alpha1 and -alpha2 releases couldn't be finished due to issues in our
release pipeline.

The most notorious changes that are worth mentioning are:

• The addition of device manager for runtime-rs
• Several improvements related to GPU usage with Kata Containers
• Several improvements to the kata-ctl tool
• Addition of artefacts and specific runtime classes for x86_64 TEEs
  • SEV, SNP, and TDX are the ones being tested, built, and shipped for now
• Multi-architecture release, including:
  • kata-static-3.2.0-alpha3-{aarch64,s390x,x86_64}.tar.xz
    • Note, aarch64 and x86_64 will be changed to arm64 and amd64 for the next
      release
  • kata-deploy payloads supporting amd64, arm64, and s390x
    • see: https://quay.io/repository/kata-containers/kata-deploy?tab=tags
• Several other bug fixes happened all over the code

Shortlog

f636c1f gha: release: Simplify the process for tagging the payload
d10c9be gha: release: login-action: Don't specify docker.io registry
0b1c5ea versions: Update nydus version to 2.2.1
eff6ed2 runtime: make debug console work with sandbox_cgroup_only
c543631 release: Kata Containers 3.2.0-alpha3
f370226 release: Fix docker/login-action version
fc09d0f release: Kata Containers 3.2.0-alpha2
4719802 runtime-rs: add virtio-blk-mmio
f9bded4 runtime-rs: add devicetype enum
6800d30 runtime-rs: remove device
f16012a runtime-rs: support linux device
fe9ec67 runtime-rs: block volume
a8bfac9 runtime-rs: support block rootfs
b076d46 agent: handle hotplug virtio-mmio device
6e273d6 runtime-rs: implement trait for vhost-user device
cc9c915 runtime-rs: implement trait for vfio device
e4c5c74 runtime-rs: device manager
22154e0 cache: Fix OVMF tarball name for different flavours
b7341cd cache: Use "initrd" as initrd_type to build rootfs-initrd
35c3d7b runtime: clh: Re-generate the client code
cfee99c versions: Upgrade to Cloud Hypervisor v32.0
b8ffcd1 osbuilder: Bump fedora image version
636539b kata-deploy: Use apt-key.gpg from k8s.io
ae24dc7 local-build: Standardise what's set for the local build scripts
ad324ad gha: aks: Wait a little bit more before run the tests
11a34a7 docs: Update container network model url
191b6dd gha: release: Fix s390x worklow
75330ab cache: Fix OVMF caching
cfd8f4f gha: payload-after-push: Pass secrets down
a89b44a tools: Fix arch bug
f527f61 release: Kata Containers 3.2.0-alpha1
ca1531f runtime: Use static_sandbox_resource_mgmt=true for TEEs
f6e1b11 agent: update tokio dependency
4cb83dc kata-ctl: update tokio dependency
df615ff runk: update tokio dependency
ca6892d runtime-rs: update tokio dependency
3e85bf5 resource-control: fix setting CPU affinities on Linux
bdb75fb runtime: use enable_vcpus_pinning from toml
fa832f4 gha: k8s: Make the tests more reliable
cbb9fe8 config: Use standard OVMF with SEV
724437e kata-deploy: add kata-qemu-sev runtimeclass
521dad2 Tests: skip CPU constraints test on SEV and SNP
72308dd gha: ci-on-push: Don't skip tests for SEV
da0f92c gha: ci-on-push: Don't skip tests for SEV-SNP
12f43be gha: tdx: Use the k3s overlay for kata-cleanup
dd75625 runtime: pkg/sev: Add kbs utility package for SEV pre-attestation
05de7b2 runtime: Add sev package
3a9d3c7 gpu: Rename the last bits from gpu to nvidia-gpu
4cde844 local-build: Fix kernel-nvidia-gpu target name
1a3f8fc deploy: fix shell script error
c5a59ca ppc64le: switch virtiofsd from C to rust version
bfdf014 versions: Bump virtiofsd to 1.6.1
87cb98c osbuilder: Fix indentation in rootfs.sh
20cb875 virtcontainers/qemu_test.go: Improve test coverage
022a33d agent: Add context to errors when AgentConfig file is missing
50cc9c5 tests: Improve coverage for virtcontainers/pkg/compatoci/ for Kata 2.0
73913c8 kata-manager: Fix '-o' syntax and logic error
593840e kata-ctl: Allow INSTALL_PATH= to be specified
5f3f844 runtime-rs: fix building instructions with respect to required Rust version
197c336 Dragonball: use LinuxBootConfigurator::write_bootparams to writes the boot parameters into guest memory.
b9a1db2 kata-deploy: Add http_proxy as part of the docker build
777c3dc kata-deploy: Do not ship the kata tarball
136e241 static-build: Download firecracker instead of building it
3bf767c static-build: Adjust ARCH for nydus
ac88d34 static-build: Use relased binary for CLH (aarch64)
2856d3f deploy: Fix arch in image tag
e8f81ee Revert "kata-deploy: Use readinessProbe to ensure everything is ready"
a4c0303 virtcontainers: Fixed static checks for improved test coverage for fc.go
03a8cd6 virtcontainers: Improved test coverage for fc.go from 4.6% to 18.5%
cfe6352 release: Fix multi-arch publishing is not supported
4d17ea4 cache: Fix nvidia-snp caching version
a133fad cache: Fix nvidia-gpu-tdx-experimental cache URL
defb643 runtime: remove overriding ARCH value by default for ppc64le
5226f15 gha: Fix Body Line Length action flagging empty body commit messages
0d49cee gha: Fix snap creation workflow warnings
b9990c2 cache: Fix nvidia-gpu version
c9bf780 cache: Update the KERNEL_FLAVOUR list to include nvidia-gpu
3665b42 gpu: Rename gpu targets to nvidia-gpu
2c90cac local-build: fixup alphabetization
4da6eb5 kata-deploy: Add qemu-snp shim
14dd053 kata-deploy: add kata-qemu-snp runtimeclass
0bb37bf config: Add SNP configuration
af7f251 versions: update SEV kernel description
dbcc3b5 local-build: fix default values for OVMF build
b8bbe63 gha: build OVMF for tests and release
cf0ca26 local-build: Add x86_64 OVMF target
db095dd cache: add SNP flavor to comments
f4ee005 gha: Build and ship QEMU for SNP
7a58a91 docs: update SNP guide
879333b versions: update SNP QEMU version
38ce4a3 local-build: add support to build QEMU for SEV-SNP
e1f3b87 docs: Mark snap installation method as unmaintained
772d4db gha: Build and ship SEV initrd
45fa366 gha: Build and ship SEV OVMF
4770d30 gha: Build and ship SEV kernel.
fb9c1fc runtime: Add qemu-sev config
813e4c5 runtimeClasses: add sev runtime class
af18806 static-build: Add caching support to sev ovmf
76ae7a3 packaging: adding caching capability for kernel
12c5ef9 packaging: add support to build OVMF for SEV
b87820e packaging: add support to build initrd for sev
b0e6a09 packaging: Add sev kernel build capability
5f8008b kata-ctl: add unit test for kvm check
a085a6d kata-ctl: add generic kvm check
6594a93 tools: made log-parser-rs
17daeb9 warning_fix: fix warnings when build with cargo-1.68.0
8495f83 cross-compile: Include documentation and configuration for cross-compile
205909f runtime: Fix virtiofs fd leak
13d7f39 gpu: Check for VFIO port assignments
138ada0 gpu: Cold Plug VFIO toml setting
f7ad75c gpu: Cold-plug extend the api.md
0fec2e6 gpu: Add cold-plug test
dded731 gpu: Add OVMF setting for MMIO aperture
2a83017 gpu: Add fwcfg helper function
131f056 gpu: Extract VFIO Functions to drivers
c8cf7ed gpu: Add ColdPlug of VFIO devices with devManager
e2b5e7f gpu: Add Rawdevices to hypervisor
6107c32 gpu: Assign default value to cold-plug
377ebc2 gpu: Add configuration option for cold-plug VFIO
c18ceae gpu: Add new struct PCIePort
1c1ee80 pkg/signals: Improved test coverage 60% to 100%
9c38204 virtcontainers/persist: Improved test coverage 65% to 87.5%
0f45b0f virtcontainers/clh_test.go: improve unit test coverage
6bf1fc6 virtcontainers/factory: Improved test coverage
5c9246d gha: Also run k8s tests on qemu-snp
c57a444 gha: Add the ability to test qemu-snp
9e2b7ff gha: sev: fix for kata-deploy error
c849bdb gha: Also run k8s tests on qemu-sev
521519d gha: Add the ability to test qemu-sev
4064192 env: Utilize arch specific functionality to get cpu details
fb40c71 env: Check for root privileges
1016bc1 config: Add api to fetch config from default config path
b908a78 kata-env: Pass cmd option for file path
b192019 config: Workaround the way agent and hypervisor configs are fetched
f2b2621 kata-env: Implement the kata-env command.
f2ebdd8 utils: Get rid of spurious print statement left behind.
9a94f1f make: Export VERSION and COMMIT
2f81f48 config: Add file under /opt as another location to look for the config
07f7d17 config: Make the pipe_size field optional
68f6357 config: Make function to get the default conf file public
7565b33 kata-ctl: Implement Display trait for GuestProtection enum
94a00f9 utils: Make certain constants in utils.rs public
572b338 gitignore: Ignore .swp and .swo editor backup files
376884b cargo: Update version of clap to 4.1.13
cc8ea32 runtime-rs: support keep_abnormal in toml config
b1730e4 gpu: Add new kernel build option to usage()
825e769 gpu: Add GPU support to default kernel without any TEE
e4ee07f gpu: Add GPU TDX experimental kernel
87ea43c gpu: Add configuration fragment
aca6ff7 gpu: Build and Ship an GPU enabled Kernel
e4b3b08 gpu: Add proper CONFIG_LOCALVERSION depending on TEE
432d407 kata-ctl: checks for kvm, kvm_intel modules loaded
3e7b902 osbuilder: Fix D-Bus enabling in the dracut case
6d31571 snap: fix docker start fail issue
96e8470 kata-manager: Fix containerd download
53c749a agent: Fix ut issue caused by fd double closed
2e3f19a agent: fix clippy warnings caused by protobuf3
4849c56 agent: Fix unit test issue cuased by protobuf upgrade
0a582f7 trace-forwarder: remove unused crate protobuf
7325385 kata-ctl: remove unused crate ttrpc
76d2e30 agent-ctl: Bump ttrpc from 0.6.0 to 0.7.1
eb3d20d protocols: Add ut for Serde
59568c7 protocols: add support for Serde
a6b4d92 runtime-rs: Bump ttrpc from 0.6.0 to 0.7.1
8a...