in Hacks

pip destroys my memory

I spent about 4 hours today debugging pip which sucks up memory when using the outdated command. It’s pretty insane. In my test case with 8 med-sized packages it takes up 60mb just to see if any of them is outdated.

So I actually took a look at the debugging tool chain for Python. There are good built-in debugger called pdb which handles the basics very similar to gdb – as far as I can tell.

A cool feature is that you can write python code in the debugger and inject it into the running program. Which of course is more problematic with C binaries. So, I debugged the different steps look at the time it takes, look deeper into the functions and saw a lot of stuff which can cause the problem.

I still can’t see the main problem however I think it lies somewhere in caching websites. As far as I can see pip downloads all version of pages of a package and caches them. This could cause the problem.

Actually, while writing this article I thought that has to be an work-around. And actually there is a pretty easy one.

pip list works without problems and it lists all packages. We can remove the version numbers with sed and join all the files with tr. Now we have a list of all packages. If we run it through pip install -U we just try to upgrade all the packages.

It works wonderfully. I think I don’t debug anymore and just create an alias.

Write a Comment


This site uses Akismet to reduce spam. Learn how your comment data is processed.