Gotten burned by that. Have a python utility that calls something that's very CPU intensive (GDAL) and I literally have to wrap this python process in java for real concurrency. Not a python expert by any means but I'd love to know what other people do when this happens. I'm assuming it's something revolving around containers and messaging (blech)...
I thought Python calling a native lib was the one time you could multithread your way around the GIL if you were cpu limited? Are the GDAL bindings not releasing the lock or something?
Had a massive amount of images that I was trying to process for a specific purpose. I wound up spinning up multiple python processes within a java wrapper using concurrent linked queues. Got the job done but it's not the proudest moment of my life.
You could have done this with the multiprocessing module. It interfaces like the threading library but it's launching new processes for each "thread". I've done a shitload of image processing in Python with it. You'll end up with a 100% Python version of what you did.
Python works around this by having multithreading through multiprocessing and IPC. its made pretty easy to use with the futures builtin module but is finicky/slow/hard to support.
