The Open Source Lab (OSUOSL) and Center for Genome Research and Biocomputing (CGRB) partner with IBM and OpenPOWER in order to provide a download resources around Open-CE. Open-CE is a community driven software distribution for machine learning that runs on standard Linux platforms with NVIDIA GPU technologies. Questions and general discussions involving OSU’s builds can be directed to the Open-CE Slack channel: https://open-ce.slack.com/archives/C06DGE5GHND.

• Current release
• Previous releases

Open-CE Release 1.11.2

This is release 1.11.2 of Open Cognitive Environment (Open-CE)

Build Status:

What’s new

• Updated packages

  • langchain 0.1.13
  • langchain-community 0.0.29
  • langchain-core 0.1.35
  • langchain-text-splitters 0.0.2
  • langsmith 0.1.19
  • libmamba 1.5.7
  • libmambapy 1.5.7
  • libopenblas 0.3.26
  • libopenblas-static 0.3.26
  • mamba 1.5.7
  • onnx 1.16.0
  • openblas[-devel] 0.3.27
  • orjson 3.9.15
  • safetensors 0.4.1
  • transformers 4.38.0
  • werkzeug 3.0.3

• This release of Open-CE supports:

  • NVIDIA’s CUDA version 11.8, 12.2
  • Python 3.10, 3.11

• Important Notes:

  • ppc64le builds with CUDA are UBI 8 container image based, not UBI 9 (amd64,arm64 only)
    • Nvidia will not provide ppc64le-based UBI 8(+) images with CUDA > 12.4.1
    • See: https://hub.docker.com/r/nvidia/cuda/tags?page=1&page_size=&ordering=&name=-devel-ubi
  • CV-CUDA is disabled in DALI for ppc64le
  • Jax and Jaxlib packages not available for ppc64le CUDA

Learn more

Get information about planning, configuring, and managing Open-CE 1.9 Below:

• Planning
• System setup
• Setting up the software repository
• Creating conda environments (recommended)
• Installing the MLDL frameworks
• Uninstalling the MLDL frameworks

Planning

We recommend users use one of the listed operating systems listed below. This is a standard conda repository and can be added to any conda install. Conda must be configured to give priority to installing packages from this channel.

System setup

Open-CE can be installed and run directly on a bare-metal RHEL and Ubuntu based systems.

Supported hardware:

• IBM Power System IC922 with NVIDIA Tesla T4 GPUs
• IBM Power System AC922 with NVIDIA Tesla V100 GPUs
• x86_64 systems with NVIDIA Tesla V100 or P100 GPUs

Supported operating systems:

• ppc64le

  • Community Enterprise Operating System (CentOS) 8.1-6
  • Red Hat Enterprise Linux for POWER LE 8.1-6
  • Rocky / Alma Linux 8.1-6
  • Ubuntu 20.04.1

• x86

  • Community Enterprise Operating System (CentOS) 7.8, 8.x
  • Red Hat Enterprise Linux for POWER LE 7.8, 8.1-6, 9.0
  • Rocky / Alma Linux 8.1-6, 9.0
  • Ubuntu 20.04.1-4, 22.04.x

Note: We (CQLS & OSL) have dropped support of RHEL/CentOS 7, as we have transitioned most systems away before the EOS date.

Required 3rd party software:

• NVIDIA GPU driver 520.61.05
• CUDA version 11.8

Installing the Open-CE Repository and Frameworks

Setting up the software repository

The Open-CE MLDL packages are distributed as conda packages in an online conda repository. Conda must be configured to give priority to installing packages from this channel.

Add the Open-CE channel to the conda configuration by running the following command:

conda config --prepend channels https://ftp.osuosl.org/pub/open-ce/current/

Creating conda environments (recommended)

With conda, you can create environments that have different versions of Python or packages installed in them. Conda environments are optional but recommended. If not used, packages are installed in the default environment called base, which often has a higher risk of containing conflicting packages or dependencies. Switching between environments is called activating the environment.

The syntax to create and activate a conda environment is:

conda create --name <environment name> python=<python version>
conda activate <environment name>

Note: It is recommended that you specify the Python version when creating a new environment. If you do not specify the version, Python 3.7 is installed when any package that requires Python are installed.

The only valid Python versions with Open-CE 1.9 are Python 3.9 and 3.10.

For example, to create an environment named opence_env with Python 3.9:

conda create --name opence_env python=3.9
0conda activate opence_env

For more information on what you can do with conda environment see https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html.

Note: Open-CE should be run as a non-privileged user and not root. The Open-CE components are designed to be usable by normal users, and the pre-installed docker images provide a non-root user by default. Some of the Open-CE components will give warnings or will fail when run as root.

Installing frameworks individually

You can install the MLDL frameworks individually. The framework packages include the following versions.

Table 1. Framework packages (Open-CE 1.9.1)

With the conda environment activated, run the following command:

conda install <package name>

Uninstalling the Open-CE MLDL frameworks

Find information about uninstalling machine learning and deep learning MLDL frameworks.

The MLDL framework packages can be uninstalled individually, or you can uninstall all of the MLDL packages at the same time.

If the frameworks are installed into a separate conda environment, all of the frameworks can be removed by simply deleting the environment:

conda env remove -n <environment name>

Individual frameworks (and any packages that depend on them) can be removed by removing the individual package:

conda remove <package name>

Important: This command removes the specified packages and any packages that depend on any of the specified packages. If you want to skip this dependency checking and remove just the requested packages, add the –force option. However, this may break your environment, so use this option with caution.

Previous releases

We recommend that you install the most current release of Open-CE, however, if you have an earlier version installed, you can find information below:

Open-CE Release 1.11.0

This is release 1.11.0 of Open Cognitive Environment (Open-CE)

Build Status:

What’s new

• Updated packages

  • absl-py 2.0.0
  • apache-beam 2.53.0
  • arrow-cpp[-proc] 15.0.1
  • bazel 6.1.0
  • black 23.10.0
  • cmdstan 2.33.1
  • cmdstanpy 1.2.0
  • cudatoolkit[-dev] 12.2.0
  • cudnn 8.9.6_12.2
  • dali[-tf-plugin] 1.32.0
  • datasets 2.16.1
  • deepspeed 0.11.1
  • fsspec 2023.10.0
  • hatch-fancy-pypi-readme 23.1.0
  • horovod 0.28.1
  • huggingface_hub 0.20.0
  • java-11-openjdk 11.0.6.10
  • jax 0.4.23
  • joblib 1.3.2
  • jsonpatch 1.33
  • keras 2.14.0
  • langchain 0.1.6
  • langchain-community 0.0.19
  • langchain-core 0.1.22
  • langsmith 0.0.87
  • libnvjitlink 12.2.140
  • lightgbm[-proc] 4.2.0
  • lightning-app 2.1.3
  • lightning-cloud 0.5.57
  • lightning-fabric 2.1.3
  • lightning-utilities 0.10.0
  • mamba 1.5.6
  • nasm 2.15.05
  • nccl 2.19.3
  • onnx 1.15.0
  • onnxmltools 1.12.0
  • onnxruntime 1.16.3
  • openblas[-devel] 0.3.26
  • [py-]opencv[-proc] 4.8.1
  • packaging 23.2
  • prophet 1.1.5
  • pyarrow 15.0.1
  • pyink 23.10.0
  • pytorch[-base|-cpu] 2.1.2
  • pytorch-lighting 2.1.3
  • pytorch_geometric 2.4.0
  • pytorch_scatter 2.1.2
  • pytorch_sparse 0.6.18
  • ray 2.9.2
  • rust 1.77.0
  • rust-std-* 1.71.1
  • scikit-learn 1.3.0
  • sentencepiece 0.1.99
  • skl2onnx 1.16.0
  • sklearn-pandas 2.2.0
  • stanio 0.3.0
  • tensorboard 2.14.0
  • tensorflow 2.14.1
  • tensorflow-datasets 4.9.4
  • tensorflow-estimator 2.14.0
  • tensorflow-hub 0.15.0
  • tensorflow-io[-gcs-filesystem] 0.35.0
  • tensorflow-metadata 1.14.0
  • tensorflow-probability 0.22.1
  • tensorflow-text 2.14.0
  • tf2onnx 1.15.1
  • tiktoken 0.6.0
  • tokenizers 0.15.2
  • torchdata 0.7.1
  • torchmetrics 1.2.1
  • torchtext 0.16.2
  • torchvision 0.16.2
  • transformers 4.36.2
  • uwsgi 2.0.25.1
  • xgboost 2.0.3

• This release of Open-CE supports:

  • NVIDIA’s CUDA version 11.8, 12.2
  • Python 3.10, 3.11

• Important Notes:

  • ppc64le builds with CUDA are UBI 8 container image based, not UBI 9 (amd64,arm64 only)
    • Nvidia will not provide ppc64le-based UBI 8(+) images with CUDA > 12.4.1
    • See: https://hub.docker.com/r/nvidia/cuda/tags?page=1&page_size=&ordering=&name=-devel-ubi
  • CV-CUDA is disabled in DALI for ppc64le
  • Jax and Jaxlib packages not available for ppc64le CUDA
  • Python 3.9 is no longer supported
  • OSU drops support of EL7

Open-CE Release 1.7.2

Release date: 09/29/2022

This is bug fix release 2 of release 1.7

What’s new

• Various build fixed
• Updated packages
  • TensorFlow 2.9.2
  • xgboost 1.6.2
  • DALI 1.16.1
  • Ray 1.13.1
  • PyTorch Geometric 2.1.0
  • numba 0.56.1
  • snapml 1.8.10
  • TF Serving 2.9.2

Learn more

Get information about planning, configuring, and managing Open-CE 1.7 Below:

• Planning
• System setup
• Setting up the software repository
• Creating conda environments (recommended)
• Installing the MLDL frameworks
• Uninstalling the MLDL frameworks

Planning

We recommend users use one of the listed operating systems listed below. This is a standard conda repository and can be added to any conda install. Conda must be configured to give priority to installing packages from this channel.

System setup

Open-CE can be installed and run directly on a bare-metal RHEL and Ubuntu based systems.

Supported hardware:

• IBM Power System IC922 with NVIDIA Tesla T4 GPUs
• IBM Power System AC922 with NVIDIA Tesla V100 GPUs
• IBM Power System S822LC with NVIDIA Tesla P100 GPUs
• x86_64 systems with NVIDIA Tesla V100 or P100 GPUs

Supported operating systems:

• Red Hat Enterprise Linux for POWER LE 7.8, 8.x
• Community Enterprise Operating System (CentOS) 7.8, 8.x
• Ubuntu 20.04.1

Required 3rd party software:

• NVIDIA GPU driver 440.33 - 520.61
• CUDA driver 11.2 or 11.4

Installing the Open-CE Repository and Frameworks

Setting up the software repository

The Open-CE MLDL packages are distributed as conda packages in an online conda repository. Conda must be configured to give priority to installing packages from this channel.

Add the Open-CE channel to the conda configuration by running the following command:

conda config --prepend channels https://ftp.osuosl.org/pub/open-ce/current/

Creating conda environments (recommended)

With conda, you can create environments that have different versions of Python or packages installed in them. Conda environments are optional but recommended. If not used, packages are installed in the default environment called base, which often has a higher risk of containing conflicting packages or dependencies. Switching between environments is called activating the environment.

The syntax to create and activate a conda environment is:

conda create --name <environment name> python=<python version> conda activate <environment name>

Note: It is recommended that you specify the Python version when creating a new environment. If you do not specify the version, Python 3.7 is installed when any package that requires Python are installed.

The only valid Python versions with Open-CE are Python 3.8, 3.9 and 3.10

For example, to create an environment named opence_env with Python 3.9:

conda create --name opence_env python=3.9 conda activate opence_env

For more information on what you can do with conda environment see https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html.

Note: Open-CE should be run as a non-privileged user and not root. The Open-CE components are designed to be usable by normal users, and the pre-installed docker images provide a non-root user by default. Some of the Open-CE components will give warnings or will fail when run as root.

Installing frameworks individually

You can install the MLDL frameworks individually. The framework packages include the following versions.

Table 1. Framework packages (Open-CE 1.7.2)

With the conda environment activated, run the following command:

conda install <package name>

Uninstalling the MLDL frameworks:

Find information about uninstalling machine learning and deep learning MLDL frameworks.

The MLDL framework packages can be uninstalled individually, or you can uninstall all of the MLDL packages at the same time.

If the frameworks are installed into a separate conda environment, all of the frameworks can be removed by simply deleting the environment:

conda env remove -n <environment name>

Individual frameworks (and any packages that depend on them) can be removed by removing the individual package:

conda remove <package name>

Important: This command removes the specified packages and any packages that depend on any of the specified packages. If you want to skip this dependency checking and remove just the requested packages, add the –force option. However, this may break your environment, so use this option with caution.

Open-CE Release 1.9.3

Release date: 12/20/2023

This is bug fix release 3 of release 1.9. No other additions have been made since 1.9.1.

What’s new

• Various bugs fixed

• Updated packages

  • xgboost 1.7.6
  • DALI 1.26
  • mamba 1.4.2
  • Onnxruntime 1.15.1
  • Pytorch 2.0.1
  • Ray 2.5.0
  • Tensorboard 2.12.2
  • Tensorflow-addons 0.19.0
  • Tensorflow Serving 2.12.1
  • Apache-beam 2.48.0

• This release of Open-CE supports:

  • NVIDIA’s CUDA version 11.8
  • Python 3.9 and 3.10

• All the packages are built with openssl 1.*.

Open-CE Release 1.6.1

Release date: 05/19/2022

This is bug fix release 1 of release 1.6

What’s new

• Various build fixed
• Updated packages
  • pytorch-lightning 1.6.3
  • pyDeprecate 0.3.2
  • torchmetrics 0.8.2
  • tensorflow-io-gcs-filesystem 0.25.0
  • ray 1.11.1

Open-CE Release 1.5.1

Release date: 01/11/2021

This is bug fix release 1 of release 1.5

What’s new

Key changes include:

Refresh PyTorch to v1.10.1 remove py36 blocks and dataclasses from all recipes Update DALI to 1.9 (from 1.9-dev) Update tensorflow metadata to 1.5.0 Enable uwsgi for python version 3.9

Open-CE Release 1.5.0

Release date: 12/08/2021

What’s new

This is release 1.5.0 of the Open Cognitive Environment (Open-CE), codenamed Otter

This release of Open-CE supports NVIDIA’s CUDA versions 10.2,11.2 as well as Python 3.7,3.8,3.9.

Open-CE Release 1.4.1

Release date: 10/10/2021

What’s new

This is bug fix 1 of release 1.4 of Open Cognitive Environment (Open-CE). Main updates are:

• TensorFlow is now at 2.6.2
• PyTorch is now at 1.9.1
• The DALI recipe now builds on both x86 and ppc.
• Bug Fix Changes
• Changes For open-ce
• Release updates for 1.4.1 (#545)
• Use updated uwsgi 2.0.20 from conda-forge (#544)
• Pin updates for 1.4.1 (#540)
• Update OpenCV to v3.4.16 (#open-ce/opencv-feedstock#27)
• Update Tensorflow Probability to v0.14.1 (#open-ce/tensorflow-probability-feedstock#19)
• Update pytorch-lightning to 1.4.9 and torchmetrics to v0.5.1 (#open-ce/pytorch-lightning-feedstock#24)

For a complete list of changes also see the 1.4.0 release.

Open-CE Release 1.3.1

Release date: 08/26/2021

What’s new

This is bug fix 1 of release 1.3 of Open Cognitive Environment (Open-CE), code named Chipmunk. Bug Fix Changes

• Fix uwsgi build #470 #474
• Adjust h5py pins for py39 #473 #482
• enable open-cv build directly in opence-env.yaml #477
• Move feedstock patches directory into /envs #484
• Update OpenBLAS to 0.3.13 #479
• Add pin for ICU #493
• adjust build resources for TensorFlow builds open-ce/tensorflow-feedstock#58 open-ce/tensorflow-feedstock#59
• TensorFlow: update to 2.5.1 open-ce/tensorflow-feedstock#61
• Pytorch: use TBB for CPU and OpenMP for GPU open-ce/pytorch-feedstock#68
• Horovod: use system compilers when using system MPI open-ce/horovod-feedstock#28
• LightGBM: use system compilers when using system MPI open-ce/LightGBM-feedstock#21
• OpenCV: disable LAPACK temporarily open-ce/opencv-feedstock#19

For a complete list of changes also see the 1.3.0 release.

Open-CE Release 1.2.2

Release date: 06/16/2021

What’s new

This is release 1.2.2 of Open Cognitive Environment (Open-CE).

This is bug fix 2 of release 1.2 of Open Cognitive Environment (Open-CE), code named Prairiedog.

Bug Fix Changes

• libgcc and libstdc++ were pinned to cos6 versions to allow for compilation with GCC 7.2/7.3 #433
• TensorFlow was updated to version 2.4.2
• Dependency pins were loosened for networkx, requests, scipy and werkzeug #439
• Changed PyArrow to build with -O2 optimizations to avoid a compiler error in GCC 7.x
• Add patch to PyArrow to fix handling of decimal types with negative scale in C data import

Previously, the Open-CE build tools were part of the Open-CE repository. They can now be found in their own repo.

A release of Open-CE now only includes:

• The Open-CE env files used to generate a conda channel containing all of the packages that are part of an Open-CE release.
• A collection of feedstocks containing conda recipes for building the packages that are part of an Open-CE release.

New Features

• PyArrow is now included as part of Open-CE.
• The protobuf version that all Open-CE packages use is now set to 3.11.2.
• TensorFlow serving was removed, due to its incompatibility with protobuf 3.11.2

Bug Fix Changes

• The conda hash string has been removed from the name of all noarch packages.
• The version of sqlite that TensorFlow uses is now explicitly set 38 39.

Open-CE is distributed as prebuilt containers, or on demand through the Conda provisioning process.

• All of the Conda packages are available in a Open-CE Conda channel
• Conda packages are available in the Open-CE 1.2.0 Conda channel
• There is no install package to download, instead connect to the Conda channel and install your packages from there
• Package dependencies are automatically resolved
• Delivery of packages is open and continuous
• Enable Python versions 3.6, 3.7, 3.8
• You can run more than one framework at the same time in the same environment. For example, you can run TensorFlow and PyTorch at the same time.

Open-CE Release 1.2.0

Release date: 04/16/2021

What’s new

This is release 1.2 of Open Cognitive Environment (Open-CE), code named Prairiedog.

Previously, the Open-CE build tools were part of the Open-CE repository. They can now be found in their own repo.

A release of Open-CE now only includes:

• The Open-CE env files used to generate a conda channel containing all of the packages that are part of an Open-CE release.
• A collection of feedstocks containing conda recipes for building the packages that are part of an Open-CE release.

New Features

• PyArrow is now included as part of Open-CE.
• The protobuf version that all Open-CE packages use is now set to 3.11.2.
• TensorFlow serving was removed, due to its incompatibility with protobuf 3.11.2

Bug Fix Changes

• The conda hash string has been removed from the name of all noarch packages.
• The version of sqlite that TensorFlow uses is now explicitly set 38 39.

Open-CE is distributed as prebuilt containers, or on demand through the Conda provisioning process.

• All of the Conda packages are available in a Open-CE Conda channel
• Conda packages are available in the Open-CE 1.2.0 Conda channel
• There is no install package to download, instead connect to the Conda channel and install your packages from there
• Package dependencies are automatically resolved
• Delivery of packages is open and continuous
• Enable Python versions 3.6, 3.7, 3.8
• You can run more than one framework at the same time in the same environment. For example, you can run TensorFlow and PyTorch at the same time.

Open-CE Release 1.1.1

Release date: 01/12/2021

What’s new

This is release 1.1 of Open Cognitive Environment (Open-CE), code named Meerkat.

• Added support for CUDA 11.0, which is currently supported on RHEL8.
• Added recipes for the following new packages: LightGBM, TensorFlow Model Optimization, TensorFlow Addons, PyTorch Lightning Bolts, Python Flatbuffers.
• Added the open-ce tool for running build and validate commands. This replaces the previously existing build_env.py and build_feedstock.py entry points to Open-CE.
• Added the open-ce test commands to test packages that are built by Open-CE. open-ce build env will now output conda environment files that can be used to create conda environments containing the packages that were just built.
• The open-ce build image command has been added to create Docker images from the output of open-ce build env.
• Open-CE build tools can now accept –cuda_versions as an argument to choose a version of CUDA to build with.
• open-ce build env will now check for circular dependencies between packages.
• open-ce build env will verify that all packages that are being built can be installed within the same conda environment before starting a build.
• Added the –skip_build_packages argument to open-ce build env.
• Jinja can now be used within any Open-CE configuration file.
• Improved performance when attempting to build packages that already exist.
• Added the patches key to the Open-CE environment files to allow for patching feedstocks.

Open-CE Release 1.0.0

Release date: 11/10/2020

What’s new

Open-CE 1.0 is the current release of Open-CE and includes the following features:

• conda packages are now available on ppc64le.
• conda packages are now available on x86.
• TensorFlow 2.3.1
• PyTorch 1.6.0
• Open-CE is distributed as prebuilt containers, or on demand through the Conda provisioning process.
  • All of the Conda packages are available in a Open-CE Conda channel
  • Conda packages are available in the Open-CE 1.0.0 Conda channel
  • There is no install package to download, instead connect to the Conda channel and install your packages from there
  • Package dependencies are automatically resolved
  • Delivery of packages is open and continuous
  • Enable Python versions 3.6, 3.7, 3.8
  • You can run more than one framework at the same time in the same environment. For example, you can run TensorFlow and PyTorch at the same time.