Wheels for PyGraphviz - advice?

[matthew-brett]

I spent some time working out how to build wheels, at least for Linux and Mac:

https://github.com/matthew-brett/pygraphviz-wheels

There are two big problems:

The tests still need the graphviz binaries installed

This is because the tests, at least, need the graphviz binaries on the path. I can imagine something that would be a lot more work, that shipped binaries in the wheel, probably renamed, to avoid clashing with possible system binaries - e.g. pygv-dot instead of dot.

Does pygraphviz do anything useful without the binaries? Does it make any sense to ship the wheels without them?

The tests are failing with graphviz installed

The tests fail differently on Mac and Linux:

• Mac (using the new graphviz installation built for the pygraphviz wheel);
• Linux (after `apt-get install graphviz).

I guess this is because of subtle differences in the output of the commands, across versions.

[dschult]

Thanks for working on this.

Pygraphviz is a wrapper around the cgraph library from GraphViz. I don't think it needs the entire GraphViz program bundled with it. But I'm not sure it is easy to pull apart the parts that it needs (the libcgraph library and the cgraph.h header file). It would be great to have it available as a wheel distribution. The question is really how much work that will be.

I haven't looked at the code/tests in a long time. It is quite possible that some rely on dot or other parts of GraphViz. I'll start looking into that and your build code now.

Thanks!

[matthew-brett]

Thanks very much for taking a look. Obviously it's no problem to rely on an installed graphviz for the tests, but I was hoping it wasn't too hard to stick to internal calls to the C libraries for the non-test code, so a wheel would be standalone and not rely on any graphviz applications.

[dschult]

(Linux build) The tests you point to are failing in many places, but the failures are all coming from an extra space added: graph { instead of graph {. I don't know where to look for a cause of this. I compile it on my machine and it doesn't do that.

(Mac Build) The tests you point to are mostly coming from graph "" { instead of graph {.

Any ideas on what package/library would control printing of spaces or empty strings? These don't cause errors on my mac installed from anaconda. Is there a good way to set up my machine to be able to recreate what travis is doing on my local machine?

[dschult]

I was mistaken earlier when I said the only libraries/files needed were ``libcgraphandcgraph.h```. We also need ```libcdt```.

[matthew-brett]

The build isn't using Anaconda - it's using Python.org Python, and packages installed from pip. Is it possible that the build is compiling with something more recent (perhaps in the dependencies) than Anaconda is using?

[dschult]

OK... first thing I notice: GraphViz v2.40 has a known bug for multigraphs that has been fixed in the latest unreleased version of Graphviz. That fix will be available in v2.42, but not v2.40. So all the tests I have running locally are with GraphViz v2.38.

Can we switch to installing GraphViz 2.38? I don't think that will affect these test failures, but its a place to start.

[matthew-brett]

Ouch - that's harder than it seems, because graphviz appears to provide only the latest release on its site, and building from a git checkout was bothersome, maybe because of old build tools on manylinux. But I'll give it a shot.

[dschult]

For NetworkX we gave up on building GraphViz from source in favor of using miniconda and its prebuilt graphviz binaries. Is that an easier route?

wget https://repo.continuum.io/miniconda/Miniconda3-4.3.21-MacOSX-x86_64.sh -O miniconda.sh
bash miniconda.sh -b -p $HOME/miniconda                                                  
export PATH="$HOME/miniconda/bin:$PATH"                                                  
hash -r
conda config --set always_yes yes --set changeps1 no                                     
conda update -q conda
# Useful for debugging any issues with conda                                             
conda info -a
conda install graphviz=2.38
dot -V

[dschult]

Making Graphviz from source worked on my Mac with git from gitlab.com. Can't find any version 2.38 tag from gitlab.com though....

git clone https://gitlab.com/graphviz/graphviz.git graphviz
cd graphviz
./autogen.sh
./configure
./make    #  or   cd lib/cdt ; make ; cd ../cgraph ; make    # if you only want those two libraries

[matthew-brett]

Yes, it works fine on Mac. The problem is with Manylinux, where the autoconf / automake / libtool versions are old.

[matthew-brett]

Is there a good known commit, on graphviz master, that we could use instead?

[jarrodmillman]

It looks like linux from scratch has a copy of 2.38:
http://ftp.oregonstate.edu/.2/lfs-website/blfs/view/svn/general/graphviz.html

And this may be helpful:
https://gitlab.com/graphviz/graphviz/issues/1350

And this:
https://gitlab.com/graphviz/graphviz/issues/1371

It seems like commit f54ac2c9 may correspond to the 2.38 release.

[matthew-brett]

OK - I think that's right, that commit f54ac2c9 is 2.38. I got graphviz building from the repo, with a bit of fiddling, and the tests pass now, on macOS and Linux, as long as graphviz is installed:

https://travis-ci.org/matthew-brett/pygraphviz-wheels/builds/416335635

Now the easy stuff is done :) - the question is whether it is possible to use or even test pygraphviz without the graphviz binaries installed.

[dschult]

Nice work!

Because pygraphviz is essentially a wrapper around the libraries I don't think it can be used or tested easily without some sort of binaries present. That said, if libcgraph and libcdt are present we should be able to test it by creating a graph, adding an edge/node and then reporting the edges and nodes of that graph. I'm not sure I understand which question we need to address... my understanding is that we would need to include a version of those libraries in the wheel if we want it to work without a separate binary installation. If we want the wheel to work with a separate binary installation then we have to find that separate installation and that has caused most of the installation questions in github.

[matthew-brett]

I am pretty sure I don't understand this very well - but - the libraries will be present, inside the wheel - of course they have to be, in order for the extension symbols to be available. But I guess that isn't what you meant?

I don't know pygraphviz at all - but to make the question more specific - is there any prospect of making some or all the examples work, just by calls into the extensions, and therefore to the embedded libraries?

https://pygraphviz.github.io/examples.html

Specifically, is it possible to make some of the examples work without any of the graphviz application binaries (like dot) on the system PATH?

[matthew-brett]

Sorry - to be specific - the auditwheel and delocate utilities find the dynamic libraries that a Python extension needs, and embed them into the wheel. So, no need to find the libraries - they will be there, but I think we currently do need to find the application binaries - like dot.

[dschult]

The code itself works without the GraphViz application binaries like dot. I guess that you are asking about the examples. The two examples in the link you show above work without dot.
Most of the other examples only use dot or neato or circo in the last 2-3 lines. They first create a graph, then create a dot file from that graph, then construct the layout and draw it using one of the GraphViz tools.

We could remove the drawing portions of those examples to make them testable. It does take away from the usefulness of the examples though. Perhaps we could leave that code as comments and then include one or two examples where those features are used more heavily (and not test those).

Am I finally getting your questions?

[matthew-brett]

You are getting my questions - thanks for bearing with me.

The reason I asked about using the application binaries, was the failures here:

https://travis-ci.org/matthew-brett/pygraphviz-wheels/jobs/414015655

They are of form:

======================================================================
ERROR: pygraphviz.tests.test_attribute_defaults.test_default_attributes
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/venv/local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/venv/local/lib/python2.7/site-packages/pygraphviz/tests/test_attribute_defaults.py", line 16, in test_default_attributes
    assert_equal(A.string().expandtabs(2),
  File "/venv/local/lib/python2.7/site-packages/pygraphviz/agraph.py", line 1278, in string
    return self.string_nop()
  File "/venv/local/lib/python2.7/site-packages/pygraphviz/agraph.py", line 1258, in string_nop
    return self.draw(format='dot', prog='nop').decode(self.encoding)
  File "/venv/local/lib/python2.7/site-packages/pygraphviz/agraph.py", line 1515, in draw
    data = self._run_prog(prog, args)
  File "/venv/local/lib/python2.7/site-packages/pygraphviz/agraph.py", line 1336, in _run_prog
    runprog = r'"%s"' % self._get_prog(prog)
  File "/venv/local/lib/python2.7/site-packages/pygraphviz/agraph.py", line 1323, in _get_prog
    raise ValueError("Program %s not found in path." % prog)
ValueError: Program nop not found in path.

Tracking through that module, the _run_prog method is used to call

• unflatten, tred, acyclic in the unflatten, tred, acyclic methods, respectively;
• any of neato, dot, twopi, circo, fdp, nop in the layout and draw methods.

Is there any way of replacing these with library calls, do you think?

[dschult]

Ahhh.... I see what you are saying.

Yes, there are some functions which use _run_prog with pipes to run these GraphViz programs using the dot files produced by pygraphviz. These are convenience functions in the sense that someone could "easily" do the same thing at the command line, but this sets up the pipes and does it all within the Python environment.

The answer (as I understand it) unfortunately is that it is essentially impossible to replace those calls with library calls. GraphViz doesn't provide that kind of interaction. There is no API for direct library calls with GraphViz.

So... it sounds like we have tools that will extract which libraries are used and include them with the wheel, but we don't have tools to automatically extract which command line tools we need available so they could be included in the wheel.

This helps me realize that while these are convenience functions, their existence means we probably should include the command line tools in the wheel too.... and that leads me to question whether we should be doing this as we'll be essentially including all of GraphViz in the wheel... rats

[matthew-brett]

I'm sorry to be slow - but just to take an example. Let's say I want to unflatten my graph. I think you are saying there is no way to take the in-memory graph contained in an AGraph instance, and unflatten it, without calling into the unflatten binary? I guess the unflatten binary does call into the library? Or is that not true ? Maybe the application binaries have a lot of code that is not in the library?

[dschult]

Yes, my understanding is that the applications binaries are primarily not using the libraries.

[matthew-brett]

A quick scan:

• nop doesn't seem to have anything big in it;
• unflatten, acyclic, do have some code, but it looks like this could fairly easily be moved out into separate files;
• tred has a little more code.

I couldn't immediately see where neato, dot etc were coming from.

I guess for now we could build the wheel, then say:

In order for this package to be fully functional, you should:

• Linux: apt-get / yum install graphviz
• macOS brew install graphviz
• Windows: install via https://graphviz.gitlab.io/_pages/Download/Download_windows.html (and we make some effort to find where the binaries went).

In the longer term, maybe we could ask the graphviz developers to refactor the application stuff out into libraries.

[dschult]

Here is a pdf (https://www.graphviz.org/pdf/libguide.pdf) that describes something similar to what you want to do: use GraphViz as a library.
I did not know about this capability. I'm not sure how easy it would be to rewrite the relevant parts of pygraphviz and I certainly don't have time to do that right now. Perhaps we can do as you suggest for a wheel with instructions for how to install graphviz. And then start the process of rewriting it to use the libraries.

But maybe I should dive into this a little more to see how much work is actually involved.

[matthew-brett]

From the PDF, it looks as though at least the layout commands can be done from the library (section 3) and maybe the drawing (2.3). That's just at a quick read though. Maybe worth contacting the maintainers with the question?

[dschult]

OK... I've looked into this more and I think it is possible to:

• Use SWIG to make the graphviz calls gvRender and gvLayout accessible from python.
• Rewrite the pygraphviz code to use those libraries instead of using the command line tools.

This would be a fairly major upgrade/change in the code base. The tests would change too.
The benefit would be ability to easily include just the libraries in a wheel distribution instead of all GraphViz.

@jarrodmillman and @matthew-brett, this will not be able to be done in a timely manner. What do you think about building a wheel that doesn't include GraphViz and telling people they need to have the GraphViz tools on the os path?

[jarrodmillman]

For now, we will release 1.5 without rewriting to use the library. Perhaps, we could rewrite before releasing 1.6. There is still some work needed to get the Windows wheels. Once that is done, I will release 1.5 (hopefully, before Monday 8/27).

[jarrodmillman]

@matthew-brett, @dschult I am starting to work on making a new pygraphviz release. It would really like to see if we can get a binary wheel for this release.

[abitrolly]

This thread is rather long. Does it makes sense to reticket it with actual status now that 3.0.0 is released?

https://gitlab.com/graphviz/graphviz/-/blob/main/CHANGELOG.md#300-2022-02-26

If the main concern is security, then for things to improve, a wheel should exist. Then budgets can be assigned, and bounty hunters can actually start improving things.

[jarrodmillman]

I am not aware of any security issues. Could you explain what you mean?

The reason we want wheels is so that installation is easier. In particular, so users don't need to install graphviz and let pygraphviz know where is located.

I took a look at the 3.0.0 release notes and didn't see any mention of the library interface being stable now. Is it no longer considered experimental? If so, could you point us to where that is stated.

If you have funding for this, please let us know. This is a volunteer project.

[abitrolly]

@jarrodmillman in your comment from September there is a statement that upstream recommends using command line tools, because they believe the GraphViz is not memory safe #167 (comment) That's a security problem.

[abitrolly]

Funding for security problems comes from different sources. If there is pygraphviz wheel, and then Google uses it in own notebooks, and then find that there are security issues, then programs like https://sos.dev/ will apply to it. GraphViz can also apply to https://github.com/PlaintextGroup/oss-virtual-incubator for funding. And at least the project should add https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/displaying-a-sponsor-button-in-your-repository to indicate it needs financial support.

[abitrolly]

Opening an account on https://opencollective.com/ and structuring the things that need financing in projects will also help to raise the transparency and build up trust for funders. Yes it requires some work that it is not writing code, but it is at least give people a way for action.

[rossbar]

hey believe the GraphViz is not memory safe #167 (comment) That's a security problem.

Ultimately, it would be a huge improvement for pygraphviz to be able to rely on the Graphviz library interface. AIUI that's the main remaining blocker for being able to get wheels out (there are hopefully more options for working around problems with the test suite).

Maybe the situation has changed on Graphviz's end, but as noted there was no information in 3.0 release re: the graphviz library interface (it'd be great to ping the graphviz devs for an update though, in case there has been progress that wasn't captured in the release notes).

Ultimately, I think it might be most productive to reiterate the above comments on the graphviz issue tracker. Pygraphviz already has code in place for using the Graphviz library interface:

pygraphviz/pygraphviz/agraph.py

Lines 1483 to 1524 in 17ba035

	def _layout(self, prog="neato", args=""):
	"""Assign positions to nodes in graph.
	.. caution:: EXPERIMENTAL
	This version of the layout command uses libgvc for layout instead
	of command line GraphViz tools like in versions <1.6 and the default.
	Optional prog=['neato'|'dot'|'twopi'|'circo'|'fdp'|'nop']
	will use specified graphviz layout method.
	>>> import pygraphviz as pgv
	>>> A = pgv.AGraph()
	>>> A.add_edge(1, 2)
	>>> A.layout()
	>>> A.layout(prog="neato", args="-Nshape=box -Efontsize=8")
	Use keyword args to add additional arguments to graphviz programs.
	The layout might take a long time on large graphs.
	Note: attaching positions in the AGraph usually doesn't affect the
	next rendering. The positions are recomputed. But if you use prog="nop"
	when rendering, it will take node positions from the AGraph attributes.
	If you use prog="nop2" it will take node and edge positions from the
	AGraph when rendering.
	"""
	_, prog = self._manually_parse_args(args, None, prog)
	# convert input strings to type bytes (encode it)
	if isinstance(prog, str):
	prog = prog.encode(self.encoding)
	gvc = gv.gvContext()
	gv.gvLayout(gvc, self.handle, prog)
	gv.gvRender(gvc, self.handle, format=b"dot", out=None)
	gv.gvFreeLayout(gvc, self.handle)
	gv.gvFreeContext(gvc)
	self.has_layout = True
	return

so in principle the building blocks are in place here as soon as the upstream project gives the green light for the library interface.

[magjac]

I'm one of the Graphviz maintainers. Things have improved somewhat, but there's nothing special about the 3.0.0 release. Just a few minor breaking changes to the API that most people probably won't notice.

Perhaps @Smattr can shed more light. He is doing the hard work of gradually improving the code in this respect.

[Smattr]

Sorry, late to the party here but I've just read through the history of this issue… The title does not make a lot of sense to me (it's a request for a WHL or a bug report about an existing WHL…?) but it seems conversation has diverged from the original topic anyway, so I'll try to respond to some things…

First off, it's surprising to me that installing pygraphviz and then exec-ing an arbitrary version of a binary is expected to work. I was under the impression the norm was to bundle/build your binary dependencies inside your WHL. Isn't that what things like Numpy do? (disclaimer: I know very little about Python packaging; the preceding may be laughably incorrect).

my understanding is that the applications binaries are primarily not using the libraries.

I don't see any Graphviz capability to run tred, acyclic, unflatten and some others via a library.

Most of the Graphviz binaries are the classic unix thin-bin/flat-lib approach where almost all functionality lives in the library. If there is a Graphviz binary whose functionality is not exposed programmatically and you would like it to be, please let us know.

If you are still interested in Graphviz version 2.38…

Differences between 2.38 and e.g. 2.42 should be minor. We've recently lost the ability to build 2.38, but a hero archaeologist could probably resurrect the ancient environment required to compile Graphviz 2.38. As always, the source remains eternal.

“This is unlikely to be a response you want to hear, but my recommendation is to continue to call command line tools…”

I think this is a quote from me, but if not I endorse it anyway. Basically YMMV. But a wise downstream greps their upstream’s bug tracker for things like “segfault” and “fuzzer”…

As I mention we are waiting for the graphviz developers to make graphviz work reliably as a library.

You mean you’re waiting on memory safety in a large ~40 year old C code base…? I realize this comment came from Jarrod, who knows more than most what we’re dealing with. I just want to temper the peanut gallery’s expectations.

That's a security problem…

Let me be clear… Do not run Graphviz in a security-sensitive context. <2 mins browsing the Graphviz issue tracker will yield you an RCE. There are multiple known existing stack- and heap-corruption vulnerabilities. I say this not to throw anyone under the bus, but to appeal to the same common sense that motivates you to look both ways before crossing the road. Graphviz was not intended to be run in a security-critical environment and it is primarily written in pre-ISO C. Money won’t fix this. @abitrolly if you want to improve Graphviz security, please try fixing some of the open bugs. As with every single open source project, we have too few hands and too much work.
</soap box>

---

Not sure what next steps are, but feel free to tag me in anything/everything. If I can be so bold as to speak for the Graphviz maintainers collectively, we are eager and willing to help/enable whatever pygraphviz is up to.

[abitrolly]

@abitrolly if you want to improve Graphviz security, please try fixing some of the open bugs. As with every single open source project, we have too few hands and too much work.

My point is that security concerns should not stop Python wheels from appearing. As HeartBleed and log4j had shown in real world security hardening better be solved by those with ambitions for that task. And I know that after log4j there has been an urge to put more money into funding those issues. I think an official warning in the docs like this one is below is enough.

Graphviz was not intended to be run in a security-critical environment and it is primarily written in pre-ISO C.

If you can point me to the compiled binaries, I can pack them into wheels for Linux. I've already done it in https://github.com/abitrolly/ksykaitai (although hadn't found the time to release the package).

[dschult]

Thanks @Smattr !! That is helpful to know.

We currently use the cgraph library to create dot files and then run those dot-files through command-line utilities to get e.g. the positions from the layout. It would be great to be able to use direct API-calls to get the layout information rather than relying on the command line interface. To make Python wheels work nicely we would need to avoid the command line interface (as I understand it).

You state that

 `Most of the Graphviz binaries are the classic unix thin-bin/flat-lib approach where almost all functionality lives in the library.`

Does "in the library" mean there is a programmatic API to get the layouts? Or does it mean that the command line tools are provided by the library?

We had an Issue on the Graphviz gitlab site about a year ago, and it has been closed. Perhaps we need to open a new one. Or perhaps we need to find a way to "build wheels" that allow for command line interaction with the contents of the wheel (do we include PowerShell in the wheel for windows os?). But it'd be better/cleaner/nicer to have Graphviz provide what is available on the command line through API calls to e.g. the cgraph library.

[Smattr]

 `Most of the Graphviz binaries are the classic unix thin-bin/flat-lib approach where almost all functionality lives in the library.`

Does "in the library" mean there is a programmatic API to get the layouts? Or does it mean that the command line tools are provided by the library?

Sorry I realize I typoed that. I meant thin-bin/fat-lib. Most of the functionality lives in the libraries, with the command line tools as typically a main and not much beyond that. However the libraries often only have the API that was necessary to enable the existing command line tools.

We had an Issue on the Graphviz gitlab site about a year ago, and it has been closed.

The title of this issue is “Build GraphViz in Windows for CI” and Graphviz is already built on Windows in CI. Though this issue actually seems to be about wanting to build a subset of Graphviz.

Graphviz has three parallel build systems: Autotools, MS Build, CMake. We've been trying to unify everything into CMake, but this effort has been ongoing for literally years. On Windows you can choose MS Build or CMake, though some components haven't yet been integrated into CMake. Either of these build systems should let you selectively build only the libraries you want.

[rossbar]

Thanks for the updates @Smattr , this is all really helpful. I'll admit that I never really pursued Graphviz's library interface because I had some nebulous idea from a github/gitlab discussion somewhere that it was a bad idea. It sounds like that impression of mine was incorrect, and that there is not inherent blocker to using library calls instead of the CLI. My impression from the above and related discussions is that there's no time like the present to try the library interface!

It seems like there's a relatively straightforward (famous last words) path forward here:

1. replace the current AGraph layout method with the private _layout which uses the Graphviz lib interface
2. Audit the rest of pygraphviz to verify that there are no other places where we explicitly call out to the Graphviz CLI
3. Try building wheels - I'd start with linux first and try to follow the auditwheel approach from the OP. Start by working locally and fix whatever remaining test issues there are. Repeat for MacOS (and windows, but I'd push that off for last, if only because I'm not as familiar with the packaging tooling).
4. Assuming 3. is accomplished successfully (for linux at least), then consider how/where to actually do the wheel building. NumPy recently(-ish) started using cibuildwheel which is probably where I'd start looking.

IMO providing a pygraphviz wheel would be the single biggest improvement the project could make --- the vast majority of issues on the issue tracker are installation issues and I would estimate that the vast majority of those would be solved out-of-hand by providing a wheel. It sounds like Graphviz's library interface is mature enough (and probably was last year as well, but I had misinterpreted the caveats) to give this a try!

[flying-sheep]

I definitely second cibuildwheel, it should be the least painful way for sure!

[rossbar]

A quick update from having looked at this recently: Graphviz's library interface for the layout+rendering functions should make it possible to provide wheels with the layout functionality encapsulated. There are still some functions that don't have a library interface e.g. acyclic, tred, unflatten which would still require a working Graphviz installation + CLI in order to work. xref the following gitlab issue:

https://gitlab.com/graphviz/graphviz/-/issues/2194

I still think it's worth pursuing wheels at this stage even if pygraphviz won't be able to fully encapsulate all of the currently available functionality - I suspect layout+rendering covers a significant chunk of current use-cases, so providing wheels that support it is worthwhile and would benefit many users.

[Smattr]

Is there any way of replacing these with library calls, do you think?

Well 6 years late, but https://gitlab.com/graphviz/graphviz/-/merge_requests/3511 completes the work to expose acyclic, tred, and unflatten as library calls. The changes are pretty blunt – they just move the existing binaries’ code into libcgraph – but I think they should be sufficient for now. If all goes smoothly, this should land in Graphviz 10.0.0.

[matthew-brett]

Excellent - thanks for the update!

[Stikyhooves]

Just adding a workaround I found (and not afraid to show my ignorance). I had the wheels error that others have seen. I noticed that the instructions for Manual Windows Download (on https://pygraphviz.github.io/documentation/stable/install.html ) specify install configs. - The recommended install code for Windows did not work directly for me. It failed, giving me an error.

PS C:\> python -m pip install --use-pep517 `
               --config-settings="--global-option=build_ext" `
               --config-settings="--global-option=-IC:\Program Files\Graphviz\include" `
               --config-settings="--global-option=-LC:\Program Files\Graphviz\lib" `
               pygraphviz

I modified the code and the install worked correctly, I no longer had the wheel issue when running the following code.

py -m pip install --use-pep517 --config-settings="--global-option=build_ext"  --config-settings="--global-option=-IC:\Program Files\Graphviz\include"  --config-settings="--global-option=-LC:\Program Files\Graphviz\lib" pygraphviz

I feel this is probably a syntax issue that anyone well initiated would translate without a thought, but I thought I would share here in case