Issues while trying to use Virtual Environment

[Sharon1Levy]

Environment

• Pythonnet version: 3.0.1
• Python version: 3.10
• Operating System: Window 10
• .NET Runtime: .Net 6.0

Details

I am trying to define the python to use venv and it crashes.

The code I try to run:

Runtime.PythonDLL = @"C:\Program Files\Python310\python310.dll";
//PythonEngine.PythonHome = "";
string pathToVirtualEnv = Directory.GetCurrentDirectory() + @"\venv";

var path = Environment.GetEnvironmentVariable("PATH").TrimEnd(';');
path = string.IsNullOrEmpty(path) ? pathToVirtualEnv : path + ";" + pathToVirtualEnv;
Environment.SetEnvironmentVariable("PATH", path, EnvironmentVariableTarget.Process);
Environment.SetEnvironmentVariable("PATH", pathToVirtualEnv, EnvironmentVariableTarget.Process);
Environment.SetEnvironmentVariable("PYTHONHOME", pathToVirtualEnv, EnvironmentVariableTarget.Process);
Environment.SetEnvironmentVariable("PYTHONPATH", $"{pathToVirtualEnv}\Lib\site-packages;{pathToVirtualEnv}\Lib", EnvironmentVariableTarget.Process);

PythonEngine.PythonPath = PythonEngine.PythonPath + ";" + $"{pathToVirtualEnv}\Lib\site-packages;{pathToVirtualEnv}\Lib";
PythonEngine.PythonHome = pathToVirtualEnv;

PythonEngine.Initialize();

---

it crashes in the last command.
the only way it doesnt crash is if i run :

Runtime.PythonDLL = @"C:\Program Files\Python310\python310.dll";
PythonEngine.PythonHome = "";
PythonEngine.Initialize();

but then I can't define the virtual environment.
I tried several solutions in the web but none worked.

Thanks

[bcowdery]

My understanding is that the python runtime detects when it's running in a venv and thus loads libraries from lib/pythonX.Y/site-packages in the virtual environment folder. It's probably not a great idea to touch the PYTHONPATH as it leads to issues exactly like what you're seeing.

Try this:

Environment.SetEnvironmentVariable("PATH", "/work/venv/bin:" + Environment.GetEnvironmentVariable("PATH"));
Environment.SetEnvironmentVariable("VIRTUAL_ENV", "/work/venv");

// DLL is not copied to the venv folder. You still need to find the DLL in your python installation directory.
// On linux, the library is always copied to /usr/lib/ but the version number can vary by distro, most are just major versioned
Runtime.PythonDLL = "/usr/lib/libpython3.so";

I'm running pythonnet in a Docker container, and this is how I've got it working. Please note that I haven't had a chance to install any pip packages yet, so i'm not 100% sure that this will get you all the way there, but it's a start. In my test container I can launch a C# app and PythonEngine.Initialize() executes without fault. I'm able to import a few basic python packages and print OS details.

Dockerfile

FROM mcr.microsoft.com/dotnet/sdk:7.0 AS build
WORKDIR /build
COPY ./ ./
RUN set -x \
    && cd /src \
    && dotnet restore \
    && dotnet publish MyApplication.Web -c Release -o /build/out --no-restore

FROM mcr.microsoft.com/dotnet/aspnet:7.0 AS
ENV PYTHONUNBUFFERED=1
EXPOSE 5000
WORKDIR /work/app
COPY --from=build /build/out ./

RUN set -x \
    # update system to get security fixes etc
    && apk update && apk upgrade -U \
    # Install Python3
    && apk add --update --no-cache python3 py3-pip  \
    && python -m venv /work/venv \
    # cleanup
    && rm -rf /var/cache/apk/* \
 
CMD [ "dotnet", "MyApplication.Web.dll" ]

Program.cs

Environment.SetEnvironmentVariable("PATH", "/work/venv/bin:" + Environment.GetEnvironmentVariable("PATH"));
Environment.SetEnvironmentVariable("VIRTUAL_ENV", "/work/venv");

Runtime.PythonDLL = "/usr/lib/libpython3.so"; 

PythonEngine.Initialize();

dynamic sys = Py.Import("sys");
dynamic os = Py.Import("os");

Console.WriteLine("Python version: {0}", sys.version);
Console.WriteLine("Current working directory: {0}", os.getcwd());
Console.WriteLine("PythonPath: {0}", PythonEngine.PythonPath);

Edit: fixed Runtime.PythonDLL

[Sharon1Levy]

hi,
i tried modifying the code as you instrucred and run the following:

var path = $"{Directory.GetCurrentDirectory()}\venv\Scripts\python";
Environment.SetEnvironmentVariable("PATH", $"{Directory.GetCurrentDirectory()}\venv\Scripts;{Environment.GetEnvironmentVariable("PATH")}");
Environment.SetEnvironmentVariable("VIRTUAL_ENV", $"{Directory.GetCurrentDirectory()}\venv\");
//Runtime.PythonDLL = @"C:\Program Files\Python310\python310.dll";
Runtime.PythonDLL = path;
PythonEngine.Initialize();

I recieve exceptions:

DllNotFoundException: Could not load C:\Documents\Projects\HomingTracker\Git\HomingTracker\HomingTracker\HomingTracker\bin\Debug\net6.0-windows\venv\Scripts\python.

Win32Exception: The specified module could not be found.

I tried to change path to var path = $"{Directory.GetCurrentDirectory()}\venv\Scripts\python.exe" and receive the followig exceptions;

MissingMethodException: Failed to load symbol Py_IncRef.

Win32Exception: The specified procedure could not be found.

[bcowdery]

I messed up a very important piece in my original example. The python DLL needs to be set to the python installation dir - it can't use the python exe like I originally assumed.

This means that your code should look more like the following on Windows:

var osPath = Environment.GetEnvironmentVariable("PATH");

var venvPath = Path.Join(Directory.GetCurrentDirectory(), "venv/");
var binPath = Path.Join(venvPath, "Scripts"); // Scripts on Windows, bin on Linux

Environment.SetEnvironmentVariable("PATH", $"{binPath};{osPath}");
Environment.SetEnvironmentVariable("VIRTUAL_ENV", venvPath);

Runtime.PythonDLL = @"C:\Program Files\Python310\python310.dll";

PythonEngine.Initialize();

Python will look in the venv folder for Library/site-packages/ to run any modules you have installed, but the runtime itself needs to be pointed at the dll in the install path because the dll does not get copied to the virtual environment.

[yueyinqiu]

Hello, I'm not sure whether it's the same issue, but I also failed to use virtual environment, following the wiki.

Here is all of my codes, including venv creation:

using Python.Runtime;
using System.Diagnostics;

var pythonDll = new FileInfo(
    @"C:/Users/yueyinqiu/AppData/Local/Programs/Python/Python312/python312.dll");
var venv = new DirectoryInfo(@"./venv");

// create venv
Process.Start(
    Path.GetFullPath("python", pythonDll.Directory!.FullName),
    ["-m", "venv", venv.FullName]).WaitForExit();

// set environment variables
var path = Environment.GetEnvironmentVariable("PATH")!.TrimEnd(';');
path = $"{path};{venv.FullName}";
Environment.SetEnvironmentVariable("PATH", path);
Environment.SetEnvironmentVariable("PYTHONHOME", venv.FullName);

var lib = Path.GetFullPath("Lib", venv.FullName);
var sitePackages = Path.GetFullPath("site-packages", lib);
var pythonPath = $"{sitePackages};{lib}";
Environment.SetEnvironmentVariable("PYTHONPATH", pythonPath);

// update PythonDLL & PythonHome & PythonPath
Runtime.PythonDLL = pythonDll.FullName;

PythonEngine.PythonHome = venv.FullName;
PythonEngine.PythonPath = pythonPath;

PythonEngine.Initialize();

It also crashed at PythonEngine.Initialize(), with the output (exit code 1):

Python path configuration:
  PYTHONHOME = 'C:\Users\yueyinqiu\Documents\MyTemporaryFiles\ConsoleApp1\ConsoleApp1\bin\Debug\net8.0\venv'
  PYTHONPATH = (not set)
  program name = 'python'
  isolated = 0
  environment = 1
  user site = 1
  safe_path = 0
  import site = 1
  is in build tree = 0
  stdlib dir = ''
  sys._base_executable = 'C:\\Users\\yueyinqiu\\Documents\\MyTemporaryFiles\\ConsoleApp1\\ConsoleApp1\\bin\\Debug\\net8.0\\ConsoleApp1.exe'
  sys.base_prefix = ''
  sys.base_exec_prefix = ''
  sys.platlibdir = 'DLLs'
  sys.executable = 'C:\\Users\\yueyinqiu\\Documents\\MyTemporaryFiles\\ConsoleApp1\\ConsoleApp1\\bin\\Debug\\net8.0\\ConsoleApp1.exe'
  sys.prefix = ''
  sys.exec_prefix = ''
  sys.path = [
    'C:\\Users\\yueyinqiu\\Documents\\MyTemporaryFiles\\ConsoleApp1\\ConsoleApp1\\bin\\Debug\\net8.0\\venv\\Lib\\site-packages',
    'C:\\Users\\yueyinqiu\\Documents\\MyTemporaryFiles\\ConsoleApp1\\ConsoleApp1\\bin\\Debug\\net8.0\\venv\\Lib',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00003264 (most recent call first):
  <no Python frame>

---

And @bcowdery , thanks for your suggestion. However I could not find the bin directory in my virtual environment. Its structure looks like:

Perhaps I did something wrong when creating the virtual environment?

[bcowdery]

Windows vs Linux ;)

The path in the virtual environment is different depending on the OS you're working with.

• Linux = /venv/bin/
• Windows = /venv/Scripts/

I would also avoid setting PYTHON_HOME as that tends to mess things up. In my tests, whenever I touched PYTHON_HOME I got the same filesystem encoding error that you're seeing. Instead, set the VIRTUAL_ENV environment variable and that is enough to get Python to look in your venv/Lib/site-packages/ folder for installed modules.

Take a look at your virtual environment Scripts/activate.bat file. All it does is set the VIRTUAL_ENV variable - in-fact, it clears PYTHON_HOME if it is set, specifically to avoid problems like this.

[yueyinqiu]

Wow that's true for activate.bat. I've tried that, and now PythonEngine.Initialize() is taking a very very long time to run (15 minutes have passed and it's still running). I think it would be another problem :(