Repository navigation
Is it possible to run python async functions in different Haskell threads? #40
Description
As a quick and dirty test I tried the following:
runPyInMain [pymain| import asyncio|]
foo ←
runPyInMain
[pyf|
async def foo():
print("running foo")
await asyncio.sleep(10)
print("ending foo")
return foo
|]
_ ←
forkIO $
runPy
[py_|
asyncio.run(foo_hs())
|]
_ ←
forkIO $
runPy
[py_|
asyncio.run(foo_hs())
|]This causes the second thread to wait until the first thread to run its async function (as expected since it's waiting for the global lock). However, top-level async calls are not supported so I can't seem to be able to call async functions from multiple places without it blocking the global lock.
Is it possible run multiple async functions at the same time, but calling it from the Haskell side?
Activity
Shimuuar commentedon Jul 26, 2026
Right now it's not. Haskell code uses single global lock because when haskell thread is blocked on lock scheduler can run other threads on capability (OS thread). And attempting to acquire GIL will block capability.
One possible solution is to run python code in separate OS thread. Something like
runAsyncPy :: Py a -> IO aand/orrunAsyncPySTM :: Py a -> IO (STM a)which spawns new OS thread for running python code.P.S. Out pf curiosity. What are you trying to use async python in haskell for?
qwbarch commentedon Jul 26, 2026
That's what I figured, although I was trying to avoid having to spawn a new thread for each async python call.
Mainly just trying to run a bunch of I/O bound functions from python libraries. I’d prefer to interface with those functions using inline-python and make most of the calls from haskell, rather than doing most logic from within python and then returning to haskell.
Thank you so much for the advice!
Shimuuar commentedon Jul 27, 2026
I skimmed docs for asyncio.run. And it seems it does need whole OS thread for itself:
Shimuuar commentedon Aug 23, 2026
Now that #41 is merged there's support for running concurrent python. It's very experimental. @qwbarch I would love to know whether it works for you and whether you encountered any problems.
P.S. By Zeus, I got nerd-sniped.
qwbarch commentedon Sep 12, 2026
Sorry for the late response and thank you so much for all your work into this!
From what I understand I don't think it fits with my use-case unless I'm mistaken. I realized my previous example is misleading, but here is an example of what I'd ideally be able to use (assuming something like pyAsync is defined/needed):
Where I'd expect both foo and bar to run on the same event loop. In this case since
runPyAsynccreates a new OS thread per call, I'm guessing it becomes too heavy for awaiting a bunch of i/o bound functions.In this case I will probably still need to use a hand-rolled solution for the above where awaiting mutliple async functions run on the same event loop right?
Shimuuar commentedon Sep 13, 2026
Currently (I think master is pretty close to release) global lock on haskell side is removed, so event plain
runPywill execute concurrently. Main distinction isrunPyAsynccould be interrupted withcancelPyalthough I have no idea how it will interact withasyncio. Please report any bug you encounter.Problem is python evaluation is foreign call from haskell's PoV. So it will use 1 OS thread for duration of call whether it uses
runPyAsyncorrunPy. No way around this. So yes you'll probably need some way to schedule execution of task on asyncio event loop. I know very little about asyncio so I can't offer any advice hereShimuuar commentedon Oct 15, 2026
0.3.0.0 is on hackage so now it's possible to run python code concurrently. Please open issue you run into problems with it.
qwbarch commentedon Oct 15, 2026
Awesome, thank you for your hard work!