2.0.15: Many failing tests

[dvzrv]

Hi! I'm packaging raptor for Arch Linux.
To be able to run tests I have to remove man files from the test setup unfortunately:

diff -ruN a/tests/rdfxml/Makefile.am b/tests/rdfxml/Makefile.am
--- a/tests/rdfxml/Makefile.am	2013-11-25 00:44:50.000000000 +0100
+++ b/tests/rdfxml/Makefile.am	2022-02-19 16:03:26.992680028 +0100
@@ -22,15 +22,15 @@
 # 
 
 # These are errors in strict mode, warnings in lax - for now
-RDF_BAGID_TEST_FILES=ex-03.rdf ex-42.rdf ex-43.rdf ex-44.rdf ex-45.rdf
+RDF_BAGID_TEST_FILES=ex-43.rdf ex-45.rdf
 
-RDF_TEST_FILES=ex-00.rdf ex-01.rdf ex-02.rdf ex-04.rdf ex-05.rdf \
-ex-06.rdf ex-07.rdf ex-08.rdf ex-09.rdf ex-10.rdf ex-11.rdf ex-12.rdf \
-ex-13.rdf ex-14.rdf ex-15.rdf ex-16.rdf ex-17.rdf ex-18.rdf \
-ex-20.rdf ex-21.rdf ex-22.rdf ex-23.rdf ex-24.rdf ex-25.rdf ex-26.rdf \
-ex-27.rdf ex-28.rdf ex-29.rdf ex-30.rdf ex-31.rdf ex-32.rdf ex-33.rdf \
-ex-34.rdf ex-35.rdf ex-36.rdf ex-37.rdf ex-38.rdf ex-39.rdf ex-40.rdf \
-ex-41.rdf ex-46.rdf ex-47.rdf ex-48.rdf ex-49.rdf ex-51.rdf \
+RDF_TEST_FILES=ex-00.rdf ex-01.rdf ex-02.rdf ex-05.rdf \
+ex-07.rdf ex-08.rdf ex-09.rdf ex-10.rdf ex-11.rdf ex-12.rdf \
+ex-13.rdf ex-14.rdf ex-15.rdf ex-16.rdf ex-17.rdf \
+ex-20.rdf ex-21.rdf ex-23.rdf ex-24.rdf ex-25.rdf ex-26.rdf \
+ex-27.rdf ex-28.rdf ex-29.rdf ex-33.rdf \
+ex-35.rdf ex-37.rdf ex-39.rdf \
+ex-48.rdf ex-49.rdf ex-51.rdf \
 ex-53.rdf ex-54.rdf ex-56.rdf ex-57.rdf ex-58.rdf ex-61.rdf \
 ex-62.rdf ex-63.rdf \
 22-rdf-syntax-ns.rdf rdfs-namespace.rdf rdf-schema.rdf \
@@ -40,15 +40,15 @@
 #   libxml2 bug report: http://bugs.gnome.org/show_bug.cgi?id=159219
 RDF_MAYFAIL_XML_TEST_FILES=ex-55.rdf
 
-RDF_BAD_TEST_FILES=bad-00.rdf bad-01.rdf bad-02.rdf bad-03.rdf \
+RDF_BAD_TEST_FILES=bad-00.rdf bad-01.rdf bad-03.rdf \
 bad-04.rdf bad-05.rdf bad-06.rdf bad-07.rdf bad-08.rdf bad-09.rdf \
 bad-10.rdf bad-11.rdf bad-12.rdf bad-13.rdf bad-14.rdf bad-18.rdf \
 bad-19.rdf bad-20.rdf bad-21.rdf bad-22.rdf bad-23.rdf
 
 RDF_BAD_NFC_TEST_FILES=bad-15.rdf bad-16.rdf bad-17.rdf
 
-RDF_WARN_TEST_FILES=warn-00.rdf warn-02.rdf warn-04.rdf \
-warn-05.rdf warn-06.rdf warn-07.rdf \
+RDF_WARN_TEST_FILES=warn-00.rdf warn-02.rdf \
+warn-07.rdf \
 $(RDF_BAGID_TEST_FILES)
 
 RDF_ASSUME_TEST_FILES=ex-19.rdf
diff -ruN a/tests/rdfxml/Makefile.in b/tests/rdfxml/Makefile.in
--- a/tests/rdfxml/Makefile.in	2014-11-02 07:04:38.000000000 +0100
+++ b/tests/rdfxml/Makefile.in	2022-02-19 16:06:59.546806167 +0100
@@ -262,14 +262,14 @@
 top_srcdir = @top_srcdir@
 
 # These are errors in strict mode, warnings in lax - for now
-RDF_BAGID_TEST_FILES = ex-03.rdf ex-42.rdf ex-43.rdf ex-44.rdf ex-45.rdf
-RDF_TEST_FILES = ex-00.rdf ex-01.rdf ex-02.rdf ex-04.rdf ex-05.rdf \
-ex-06.rdf ex-07.rdf ex-08.rdf ex-09.rdf ex-10.rdf ex-11.rdf ex-12.rdf \
-ex-13.rdf ex-14.rdf ex-15.rdf ex-16.rdf ex-17.rdf ex-18.rdf \
-ex-20.rdf ex-21.rdf ex-22.rdf ex-23.rdf ex-24.rdf ex-25.rdf ex-26.rdf \
-ex-27.rdf ex-28.rdf ex-29.rdf ex-30.rdf ex-31.rdf ex-32.rdf ex-33.rdf \
-ex-34.rdf ex-35.rdf ex-36.rdf ex-37.rdf ex-38.rdf ex-39.rdf ex-40.rdf \
-ex-41.rdf ex-46.rdf ex-47.rdf ex-48.rdf ex-49.rdf ex-51.rdf \
+RDF_BAGID_TEST_FILES = ex-43.rdf ex-45.rdf
+RDF_TEST_FILES = ex-00.rdf ex-01.rdf ex-02.rdf ex-05.rdf \
+ex-07.rdf ex-08.rdf ex-09.rdf ex-10.rdf ex-11.rdf ex-12.rdf \
+ex-13.rdf ex-14.rdf ex-15.rdf ex-16.rdf ex-17.rdf \
+ex-20.rdf ex-21.rdf ex-23.rdf ex-24.rdf ex-25.rdf ex-26.rdf \
+ex-27.rdf ex-28.rdf ex-29.rdf ex-33.rdf \
+ex-35.rdf ex-37.rdf ex-39.rdf \
+ex-48.rdf ex-49.rdf ex-51.rdf \
 ex-53.rdf ex-54.rdf ex-56.rdf ex-57.rdf ex-58.rdf ex-61.rdf \
 ex-62.rdf ex-63.rdf \
 22-rdf-syntax-ns.rdf rdfs-namespace.rdf rdf-schema.rdf \
@@ -279,14 +279,14 @@
 # ex-55 fails with libxml2
 #   libxml2 bug report: http://bugs.gnome.org/show_bug.cgi?id=159219
 RDF_MAYFAIL_XML_TEST_FILES = ex-55.rdf
-RDF_BAD_TEST_FILES = bad-00.rdf bad-01.rdf bad-02.rdf bad-03.rdf \
+RDF_BAD_TEST_FILES = bad-00.rdf bad-01.rdf bad-03.rdf \
 bad-04.rdf bad-05.rdf bad-06.rdf bad-07.rdf bad-08.rdf bad-09.rdf \
 bad-10.rdf bad-11.rdf bad-12.rdf bad-13.rdf bad-14.rdf bad-18.rdf \
 bad-19.rdf bad-20.rdf bad-21.rdf bad-22.rdf bad-23.rdf
 
 RDF_BAD_NFC_TEST_FILES = bad-15.rdf bad-16.rdf bad-17.rdf
-RDF_WARN_TEST_FILES = warn-00.rdf warn-02.rdf warn-04.rdf \
-warn-05.rdf warn-06.rdf warn-07.rdf \
+RDF_WARN_TEST_FILES = warn-00.rdf warn-02.rdf \
+warn-07.rdf \
 $(RDF_BAGID_TEST_FILES)
 
 RDF_ASSUME_TEST_FILES = ex-19.rdf

I am not sure whether these files are still useful or are failing due to changes in the dependencies of raptor.
Having so many failing test files is not great though and eventually leads to downstreams disabling test suites altogether.

It would be amazing if you could create a new release (see #48 ) so that the failing files can subsequently be identified and fixed/removed.

[dvzrv]

Okay, scratch that. The tests are actually randomly failing no matter what I do.

This is rather bad and I will not run tests at all because of this :(

[dvzrv]

With the following I can at least disable the broken tests from the subdirs:

diff -ruN a/tests/Makefile.am b/tests/Makefile.am
--- a/tests/Makefile.am	2013-12-08 02:57:05.000000000 +0100
+++ b/tests/Makefile.am	2022-02-19 16:47:41.562672003 +0100
@@ -37,7 +37,7 @@
 # Used to make N-triples output consistent
 BASE_URI=http://librdf.org/raptor/tests/
 
-SUBDIRS = rdfxml ntriples ntriples-2013 nquads-2013 turtle turtle-2013 trig grddl rdfa rdfa11 json feeds
+SUBDIRS = ntriples ntriples-2013 nquads-2013 turtle turtle-2013 trig grddl rdfa rdfa11 json
 
 
 $(top_builddir)/src/libraptor2.la:
diff -ruN a/tests/Makefile.in b/tests/Makefile.in
--- a/tests/Makefile.in	2014-11-02 07:04:38.000000000 +0100
+++ b/tests/Makefile.in	2022-02-19 16:47:41.569338695 +0100
@@ -338,7 +338,7 @@
 
 # Used to make N-triples output consistent
 BASE_URI = http://librdf.org/raptor/tests/
-SUBDIRS = rdfxml ntriples ntriples-2013 nquads-2013 turtle turtle-2013 trig grddl rdfa rdfa11 json feeds
+SUBDIRS = ntriples ntriples-2013 nquads-2013 turtle turtle-2013 trig grddl rdfa rdfa11 json
 all: all-recursive
 
 .SUFFIXES:

Although it is not ideal, it at least allows me to run the remaining tests. 🎉

[dvzrv]

@dajobe fixing or at least acknowledging these test problems before a new release would have been really great :)

With 2.0.16 the tests are still broken and even with the above disabled tests I still have a flaky test (fails 50% of the time):

Testing Turtle parsing with N-Triples tests
Checking test-00.ttl Checking bad-00.ttl Checking test-28.ttl Testing turtle serialization with legal turtle
Checking test-00.ttl Checking ../ntriples/test.nt Testing turtle serialization with legal rdf/xml
Checking ../rdfxml/ex-00.rdf ok
FAILED
../../utils/rapper -q -i turtle -o ntriples ./test-00.ttl http://www.w3.org/2001/sw/DataAccess/df1/tests/test-00.ttl > test-00.res
ok

I will now also disable the turtle tests:

diff -ruN a/tests/Makefile.am b/tests/Makefile.am
--- a/tests/Makefile.am	2014-11-14 19:11:13.000000000 +0100
+++ b/tests/Makefile.am	2023-03-02 19:49:59.269974344 +0100
@@ -37,7 +37,7 @@
 # Used to make N-triples output consistent
 BASE_URI=http://librdf.org/raptor/tests/
 
-SUBDIRS = rdfxml ntriples ntriples-2013 nquads-2013 turtle mkr turtle-2013 trig grddl rdfa rdfa11 json feeds
+SUBDIRS = ntriples ntriples-2013 nquads-2013 mkr turtle-2013 trig grddl rdfa rdfa11 json
 
 
 $(top_builddir)/src/libraptor2.la:
diff -ruN a/tests/Makefile.in b/tests/Makefile.in
--- a/tests/Makefile.in	2023-03-01 18:58:10.000000000 +0100
+++ b/tests/Makefile.in	2023-03-02 19:50:09.006659970 +0100
@@ -618,7 +618,7 @@
 
 # Used to make N-triples output consistent
 BASE_URI = http://librdf.org/raptor/tests/
-SUBDIRS = rdfxml ntriples ntriples-2013 nquads-2013 turtle mkr turtle-2013 trig grddl rdfa rdfa11 json feeds
+SUBDIRS = ntriples ntriples-2013 nquads-2013 mkr turtle-2013 trig grddl rdfa rdfa11 json
 all: all-recursive
 
 .SUFFIXES:

[dajobe]

These are not broken for me, I test this regularly on several different linux distributions (Debian, Ubuntu, Fedora, Gentoo). I don't happen to have/use Arch.

So you'll need to expand on "broken".

Maybe try 'make -j1 check' to avoid any multi-threaded test running issues, although that seems to be ok these days, it just makes the output easier to read and diagnose.

[dvzrv]

Maybe try 'make -j1 check' to avoid any multi-threaded test running issues, although that seems to be ok these days, it just makes the output easier to read and diagnose.

Good catch! That seems to fix the tests (and I guess also explains the flakiness/ random behavior).
Do the tests share common resources (e.g. tmp dirs etc.)?

Either way, I can drop the patch and run the tests using only one job now! Thanks! 🎉