Python virtual environment in NX monorepo – Python

The Problem:

In a JavaScript project managed using NX monorepo, part of the codebase relies on an external Python project with its own virtual environment called ‘testenv’. Certain Node.js components execute Python commands and retrieve data for the build process. When running NX scripts or tests, this virtual environment needs to be activated. However, NX spawns isolated processes, causing issues with accessing the virtual environment. The goal is to find a solution to activate the ‘testenv’ virtual environment before running specific tests and set up a custom environment for these spawned processes.

The Solutions:

Solution 1: Using Custom Environment Variables

To start a Python virtual environment before specific tests and set up a custom environment for spawned processes in NX monorepo, you can follow these steps:

1. Set Environment Variable:

   • Create an environment variable called VIRTUAL_ENV that points to the bin directory of your Python virtual environment. On Windows, replace bin with Scripts.
   • For example, if your virtual environment is located at /path/to/venv, set VIRTUAL_ENV to /path/to/venv/bin.

2. Prefix Commands with Path:

   • In your NX configuration, prefix all commands that need to be executed within the virtual environment with the VIRTUAL_ENV environment variable.
   • For example, instead of using conda activate testenv, use $VIRTUAL_ENV/conda activate testenv.

Here’s an example of how to use VIRTUAL_ENV to run a Python command in an NX script:

import os
import subprocess

# Get the value of the VIRTUAL_ENV environment variable
virtual_env = os.environ.get("VIRTUAL_ENV")

# If VIRTUAL_ENV is not set, raise an error
if not virtual_env:
    raise RuntimeError("VIRTUAL_ENV environment variable is not set.")

# Construct the command to run
python_command = f"{virtual_env}/python -m pip install some_package"

# Execute the command
subprocess.run(python_command, shell=True)

By prefixing commands with VIRTUAL_ENV, you can ensure that they are executed within the correct virtual environment, even when NX spawns new processes.

Solution 2: Using `@nxlv/python:run-commands` Executor

If you are already using the Nx python plugin, you can take advantage of the @nxlv/python:run-commands executor. This executor allows you to run Python commands in a virtual environment without having to manually activate it.

Here’s an example of how you can use the @nxlv/python:run-commands executor:

{
  "virtualenv-conda": {
    "executor": "@nxlv/python:run-commands",
    "options": {
      // The path to your virtual environment's bin directory
      "binPath": "/path/to/your_virtualenv/bin",
      // The commands you want to run in the virtual environment
      "commands": ["python script.py"]
    }
  }
}

With this configuration, you can simply run nx virtualenv-conda to activate the virtual environment and run the specified commands.

Solution 2: Using `nx:run-commands` Executor with Manual Virtual Environment Activation

If you are not using the Nx python plugin or prefer not to use the @nxlv/python:run-commands executor, you can still use the nx:run-commands executor with manual virtual environment activation.

To do this, you will need to add source ./.venv/bin/activate ; to the beginning of each command you want to run in the virtual environment.

Here’s an example of how you can use the nx:run-commands executor with manual virtual environment activation:

{
  "virtualenv-conda": {
    "executor": "nx:run-commands",
    "options": {
      "commands": [
        "source ./.venv/bin/activate ; python script.py"
      ],
      "parallel": false
    }
  }
}

With this configuration, you will need to manually activate the virtual environment before running nx virtualenv-conda. You can do this by running the following command in your terminal:

source ./.venv/bin/activate

Once the virtual environment is activated, you can run nx virtualenv-conda to run the specified commands.

Video Explanation: