Problems with DEFSYSTEM-DEPENDS-ON

[Mark Evenson]

The usage of DEFSYSTEM-DEPENDS-ON to specify dependencies that will be
satisfied by QL:QUICKLOAD no longer seems to be working in asdf-3.3.1.

It used to be the case that to use the [PROVE testing framework][1], it was
sufficient to place a

…
:defsystem-depends-on (:prove-asdf)
…

clause in the secondary system to be tested, and then upon a QL:QUICKLOAD of this system,
the dependency on PROVE was then resolved via a network download from Quicklisp.


[1]: https://github.com/fukamachi/prove

I believe this was working with asdf-3.2, but testing that assumption is a little hard as
asdf-3.3 refuses to degrade itself with a downgrade, and all my systems are running asdf-3.3.

Maybe something in Quicklisp changed as well?

[Attila Lendvai]

Post by Mark Evenson
The usage of DEFSYSTEM-DEPENDS-ON to specify dependencies that will be
satisfied by QL:QUICKLOAD no longer seems to be working in asdf-3.3.1.

FTR, here's the history of this issue:

https://github.com/quicklisp/quicklisp-client/pull/122#issuecomment-160419822

https://github.com/quicklisp/quicklisp-client/issues/108

[Mark Evenson]

Post by Attila Lendvai

Post by Mark Evenson
The usage of DEFSYSTEM-DEPENDS-ON to specify dependencies that will be
satisfied by QL:QUICKLOAD no longer seems to be working in asdf-3.3.1.

https://github.com/quicklisp/quicklisp-client/pull/122#issuecomment-160419822
https://github.com/quicklisp/quicklisp-client/issues/108

Wow! Holy stale complications, batman!

Robert apparently suggested something (apparently) much simpler in

https://github.com/quicklisp/quicklisp-client/pull/128

but without any commentary from Zach on that approach.

Given asdf-3.3 is out, and recent sbcl’s ship with it, which is the preferred way forward from ASDF’s perspective?

[Robert Goldman]

Post by Mark Evenson

Post by Attila Lendvai

Post by Mark Evenson
The usage of DEFSYSTEM-DEPENDS-ON to specify dependencies that will be
satisfied by QL:QUICKLOAD no longer seems to be working in
asdf-3.3.1.

https://github.com/quicklisp/quicklisp-client/pull/122#issuecomment-160419822
https://github.com/quicklisp/quicklisp-client/issues/108

Wow! Holy stale complications, batman!
Robert apparently suggested something (apparently) much simpler in
https://github.com/quicklisp/quicklisp-client/pull/128
but without any commentary from Zach on that approach.
Given asdf-3.3 is out, and recent sbcl’s ship with it, which is the
preferred way forward from ASDF’s perspective?

"From ASDF's perspective," this is all new to me, since it was filed as
a bug against Quicklisp, and as far as I know, never raised as an issue
for ASDF. I could use some help here:

1. What's a minimal error case using `quickload` alone?
2. What's a minimal case that arises with using ASDF as the entry point?
It seemed like there was one where if Quicklisp is up and running, and
you use `asdf:load-system` to load a system, this can also happen.

Something I can type into a REPL verbatim is what I would like to see.

Also, sounds like though this is an issue on all lisps, not just ABCL as
the first post suggested

Communications between ASDF and QL have been difficult since Zach
dropped off this list (and, to be fair, I have never joined up to read
quicklisp-devel, if there is such a thing).

Best,
R

[Mark Evenson]

Post by Mark Evenson
The usage of DEFSYSTEM-DEPENDS-ON to specify dependencies that will be
satisfied by QL:QUICKLOAD no longer seems to be working in asdf-3.3.1.
https://github.com/quicklisp/quicklisp-client/pull/122#issuecomment-160419822
https://github.com/quicklisp/quicklisp-client/issues/108
Wow! Holy stale complications, batman!
Robert apparently suggested something (apparently) much simpler in
https://github.com/quicklisp/quicklisp-client/pull/128
but without any commentary from Zach on that approach.
Given asdf-3.3 is out, and recent sbcl’s ship with it, which is the preferred way forward from ASDF’s perspective?
• What's a minimal error case using quickload alone?
• What's a minimal case that arises with using ASDF as the entry point?
It seemed like there was one where if Quicklisp is up and running, and you use asdf:load-system to load a system, this can also happen.
Something I can type into a REPL verbatim is what I would like to see.

Not sure how to distinguish between your two requests for quickload alone versus ASDF as an entry point

A minimal case would be the following ASDF definition

--—depends.asd---

(defsystem depends
:in-order-to ((test-op (test-op "depends/t"))))

(defsystem depends/t
:defsystem-depends-on (prove-asdf)
:depends-on (prove)
:components ((:test-file "depends-test.lisp")))

——depends-test.lisp——

(in-package :cl-user)
(prove:plan 1)
(prove:pass "A test that always passes")
(prove:finalize)

----------------------

(ql:quickload :depends) should pick up the depends/t secondary system to install PROVE from the network, which is needed to provide a CLOS for the TEST-FILE component.

Component "prove-asdf" not found, required by NIL
0: (CONDITIONS::CONDITIONS-ERROR :INVISIBLEP T ASDF/FIND-COMPONENT:MISSING-DEPENDENCY (:REQUIRED-BY NIL :REQUIRES "prove-asdf"))
1: (ERROR ASDF/FIND-COMPONENT:MISSING-DEPENDENCY :REQUIRED-BY NIL :REQUIRES "prove-asdf")
2: (ASDF/FIND-COMPONENT:RESOLVE-DEPENDENCY-NAME NIL "prove-asdf" NIL)
3: ((SUBFUNCTION 1 ASDF/PARSE-DEFSYSTEM:REGISTER-SYSTEM-DEFINITION))
…

For ASDF3 alone, as long as PROVE is installed, there is no problem.

Post by Mark Evenson
Also, sounds like though this is an issue on all lisps, not just ABCL as the first post suggested

Yes, this issue effects all Common Lisp implementations. I don’t think I even mentioned ABCL in my first message, so other than being an ABCL maintainer, I don’t see how you got that impression.

Post by Mark Evenson
Communications between ASDF and QL have been difficult since Zach dropped off this list (and, to be fair, I have never joined up to read quicklisp-devel, if there is such a thing).

Yes, we are certainly dealing with the resistance of Quicklisp to deprecate ASDF2 in favor of ASDF3, for which I neither really know nor want to go into the history thereof. Rather than pointing fingers, and spreading blame, I am trying to find some compromise that works for both the ASDF and Quicklisp maintainers, as without getting ql:quickload to somehow include :defsystem-depends-on declarations as recognized load dependencies in the currently stable ASDF3, it means this useful feature for ASDF extensiblity is effectively unusable for inter-system cooperation within Quicklisp.

In the January 2018 Quicklisp systems, there are 103 references to prove-asdf, so this issue effects quite a bit of the current Quicklisp distributed ecosystem for that use case alone.

As I read the Quicklisp issues and pull-requests, Quicklisp would be willing to accept a “minimally invasive” patch if it would support asdf-2.26 as well as ASDF3.


So, to put things more succintly, given the choice between Quicklisp pulls [122][] or [128][], and given that we have advanced to asdf-3.3.1 since these requests were issued, what would be the preferred manner to patch Quicklisp that would be the most forward-looking for future ASDF3 compatibility so that Quicklisp might continue to work with :DEFSYSTEM-DEPENDS-ON clauses like it used to?

[122]: https://github.com/quicklisp/quicklisp-client/pull/122
[128]: https://github.com/quicklisp/quicklisp-client/pull/128

[Robert Goldman]

OK, I think I understand now. This happens when we need quicklisp not
just to load, but to install and load, defsystem depends on systems.

I just looked at Robert Dodier's bug fix,
https://github.com/quicklisp/quicklisp-client/pull/128/ and I'm a little
concerned that it could raise an error if `asdf::missing-requires` isn't
implemented on the error condition that is signaled by ASDF (and I don't
see why it would be guaranteed to be implemented there). Shouldn't this
check the type of `(asdf::error-condition c)`? Or if there's something
about the conditions under which this handler is invoked that guarantees
that the call to `missing-requires` will not error out, I don't see it,
so it's probably worthy of a comment.

My guess is that if you checked for the `error-condition` being a
subtype of `missing-component` that would work in modern ASDF. I have
no idea whether it would work in ASDF 2 and I'm afraid that I don't have
the time for the ASDF archaeology required to figure out how to
"past-proof" this code.

HtH,
Best,
r

Post by Mark Evenson

Post by Mark Evenson
The usage of DEFSYSTEM-DEPENDS-ON to specify dependencies that will be
satisfied by QL:QUICKLOAD no longer seems to be working in
asdf-3.3.1.
https://github.com/quicklisp/quicklisp-client/pull/122#issuecomment-160419822
https://github.com/quicklisp/quicklisp-client/issues/108
Wow! Holy stale complications, batman!
Robert apparently suggested something (apparently) much simpler in
https://github.com/quicklisp/quicklisp-client/pull/128
but without any commentary from Zach on that approach.
Given asdf-3.3 is out, and recent sbcl’s ship with it, which is the
preferred way forward from ASDF’s perspective?
"From ASDF's perspective," this is all new to me, since it was filed
as a bug against Quicklisp, and as far as I know, never raised as an
• What's a minimal error case using quickload alone?
• What's a minimal case that arises with using ASDF as the entry
point?
It seemed like there was one where if Quicklisp is up and running,
and you use asdf:load-system to load a system, this can also happen.
Something I can type into a REPL verbatim is what I would like to see.

Not sure how to distinguish between your two requests for quickload
alone versus ASDF as an entry point
A minimal case would be the following ASDF definition
--—depends.asd---
(defsystem depends
:in-order-to ((test-op (test-op "depends/t"))))
(defsystem depends/t
:defsystem-depends-on (prove-asdf)
:depends-on (prove)
:components ((:test-file "depends-test.lisp")))
——depends-test.lisp——
(in-package :cl-user)
(prove:plan 1)
(prove:pass "A test that always passes")
(prove:finalize)
----------------------
(ql:quickload :depends) should pick up the depends/t secondary system
to install PROVE from the network, which is needed to provide a CLOS
for the TEST-FILE component.
Component "prove-asdf" not found, required by NIL
0: (CONDITIONS::CONDITIONS-ERROR :INVISIBLEP T
ASDF/FIND-COMPONENT:MISSING-DEPENDENCY (:REQUIRED-BY NIL :REQUIRES
"prove-asdf"))
1: (ERROR ASDF/FIND-COMPONENT:MISSING-DEPENDENCY :REQUIRED-BY NIL :REQUIRES "prove-asdf")
2: (ASDF/FIND-COMPONENT:RESOLVE-DEPENDENCY-NAME NIL "prove-asdf" NIL)
3: ((SUBFUNCTION 1 ASDF/PARSE-DEFSYSTEM:REGISTER-SYSTEM-DEFINITION))


For ASDF3 alone, as long as PROVE is installed, there is no problem.

Post by Mark Evenson
Also, sounds like though this is an issue on all lisps, not just ABCL
as the first post suggested

Yes, this issue effects all Common Lisp implementations. I don’t
think I even mentioned ABCL in my first message, so other than being
an ABCL maintainer, I don’t see how you got that impression.

Post by Mark Evenson
Communications between ASDF and QL have been difficult since Zach
dropped off this list (and, to be fair, I have never joined up to
read quicklisp-devel, if there is such a thing).

Yes, we are certainly dealing with the resistance of Quicklisp to
deprecate ASDF2 in favor of ASDF3, for which I neither really know nor
want to go into the history thereof. Rather than pointing fingers,
and spreading blame, I am trying to find some compromise that works
for both the ASDF and Quicklisp maintainers, as without getting
ql:quickload to somehow include :defsystem-depends-on declarations as
recognized load dependencies in the currently stable ASDF3, it means
this useful feature for ASDF extensiblity is effectively unusable for
inter-system cooperation within Quicklisp.
In the January 2018 Quicklisp systems, there are 103 references to
prove-asdf, so this issue effects quite a bit of the current Quicklisp
distributed ecosystem for that use case alone.
As I read the Quicklisp issues and pull-requests, Quicklisp would be
willing to accept a “minimally invasive” patch if it would support
asdf-2.26 as well as ASDF3.
So, to put things more succintly, given the choice between Quicklisp
pulls [122][] or [128][], and given that we have advanced to
asdf-3.3.1 since these requests were issued, what would be the
preferred manner to patch Quicklisp that would be the most
forward-looking for future ASDF3 compatibility so that Quicklisp might
continue to work with :DEFSYSTEM-DEPENDS-ON clauses like it used to?
[122]: https://github.com/quicklisp/quicklisp-client/pull/122
[128]: https://github.com/quicklisp/quicklisp-client/pull/128

[Robert Dodier]

Thanks to everyone for their interest in this topic.

In reference to
https://github.com/quicklisp/quicklisp-client/pull/128/, it is only a
minimal stop-gap measure which I found could fix the immediate problem
which I encountered. I know very little about ASDF and Quicklisp, so I
certainly had no conscious intent to assert that one condition or
another must prevail. Any improvements by more knowledgeable people
would be welcomed by me.

All the best,
Robert Dodier

[Robert Goldman]

Post by Robert Dodier
Thanks to everyone for their interest in this topic.
In reference to
https://github.com/quicklisp/quicklisp-client/pull/128/, it is only a
minimal stop-gap measure which I found could fix the immediate problem
which I encountered. I know very little about ASDF and Quicklisp, so I
certainly had no conscious intent to assert that one condition or
another must prevail. Any improvements by more knowledgeable people
would be welcomed by me.
All the best,
Robert Dodier

A cheesy fix would simply be to wrap it in `IGNORE-ERRORS`. But it
might cause errors in its present form.

This really is more a QL issue than an ASDF one (although it illustrates
an issue with wrapping errors, IMO).

Best,
r

[Attila Lendvai]

A cheesy fix would simply be to wrap it in IGNORE-ERRORS. But it might cause
errors in its present form.

i've learned, painfully, that indiscriminate ignore-errors will almost
always bite you back (in the form of wasted debugging time), no matter
how innocent they look.

This really is more a QL issue than an ASDF one (although it illustrates an
issue with wrapping errors, IMO).

while this is true, the implementation of QL requires a reliable way
to hook into the internal state of various versions of ASDF (namely,
into the situation when ASDF is looking for a system, and not finding
it will lead to an error without QL intervention). if i understand it
correctly, this is the crux of this issue.
--
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
“A general State education is a mere contrivance for molding people to
be exactly like one another; and as the mold in which it casts them is
that which pleases the dominant power in the government […]; it
establishes a despotism over the mind, leading by a natural tendency
to one over the body.”
— John Stuart Mill (1806–1873), 'On Liberty' (1869)

[Robert Goldman]

Post by Attila Lendvai

A cheesy fix would simply be to wrap it in IGNORE-ERRORS. But it might cause
errors in its present form.

i've learned, painfully, that indiscriminate ignore-errors will almost
always bite you back (in the form of wasted debugging time), no matter
how innocent they look.

This really is more a QL issue than an ASDF one (although it
illustrates an
issue with wrapping errors, IMO).

while this is true, the implementation of QL requires a reliable way
to hook into the internal state of various versions of ASDF (namely,
into the situation when ASDF is looking for a system, and not finding
it will lead to an error without QL intervention). if i understand it
correctly, this is the crux of this issue.

Sure, and I am happy to try to support this, but not to the extent of
recovering a copy of ASDF 2.x and trying to run it.

The problem is that I don't know when the missing component condition
was added to ASDF, and doing this right would involve checking the
enclosed condition to see if it's a missing component error. I know how
to do that for a modern ASDF, but I don't know how to handle ASDFs that
are too old to have this condition class. And I don't fee like it's my
job to think about that: I think it's perverse to continue trying to use
ASDF 2.

Best,
r

[Faré]

Post by Robert Goldman

Post by Attila Lendvai

A cheesy fix would simply be to wrap it in IGNORE-ERRORS. But it might cause
errors in its present form.

i've learned, painfully, that indiscriminate ignore-errors will almost
always bite you back (in the form of wasted debugging time), no matter
how innocent they look.

This really is more a QL issue than an ASDF one (although it illustrates an
issue with wrapping errors, IMO).

while this is true, the implementation of QL requires a reliable way
to hook into the internal state of various versions of ASDF (namely,
into the situation when ASDF is looking for a system, and not finding
it will lead to an error without QL intervention). if i understand it
correctly, this is the crux of this issue.

Sure, and I am happy to try to support this, but not to the extent of
recovering a copy of ASDF 2.x and trying to run it.
The problem is that I don't know when the missing component condition was
added to ASDF, and doing this right would involve checking the enclosed
condition to see if it's a missing component error. I know how to do that
for a modern ASDF, but I don't know how to handle ASDFs that are too old to
have this condition class. And I don't fee like it's my job to think about
that: I think it's perverse to continue trying to use ASDF 2.

ASDF 2.26 is totally unsupported at this point. No implementation uses
anything less than 3.1.2 (the first stable release in the 3.1 series,
from May 2014). Many essential packages require 3.1.x or later. It is
a waste of time to try to get 2.26 running

If Quicklisp wants to be conservative, I would recommend requiring ior
providing ASDF 3.1.7, which is the last in the 3.1 series and pretty
stable, from March 2016, which is two years ago.

The "official" policy of ASDF was always to not support anything that
had already been superseded 2 years ago or more. 3.1.7 should be the
oldest supported ASDF release.

That said, what my opinions seem to be negatively taken into account
by Xach, so there.

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org
Evolution competitively selects stable cooperative patterns.

[Robert Goldman]

I think Xach and I are trying to cope with essentially the same problem:
lack of resources for a community project.

On the one hand, Xach wants to stick with and old ASDF because it's a
lot of trouble to maintain it, and he's afraid of breakage.

OTOH, we don't have the resources to maintain backwards-compatibility in
ASDF (though we try not to break things gratuitously). Keeping ancient
unmaintained software running at the cost of introducing complexity into
ASDF is something I simply can't do (when I transfer maintenance to
Microsoft, they'll have that covered!).

Faré has worked heroically (with admirable assistance from Anton
Vodonosv) to keep from breaking libraries and offering patches, but at
the end of the day, if there's no one available to merge patches into
new releases of libraries, that can't be a reason to veto introduction
of a fix into ASDF. If there's no one available to change `foo-test`
into `foo/test` in `crusty-library.asd` that's not really our fault (and
really, should people be using `crusty-library.asd` if there isn't?).
Naturally, that has the potential to make Xach's life miserable. It's
not my fault or his.

When there's stuff like `crusty-library` out there, there will be a
certain amount of misery to be had, and there will always be a tension
between QL and ASDF about who gets to enjoy it.

I wish that I could coordinate more effectively with Xach, but I really
don't know what I can do to make this work better.

Best,
r

Post by Faré

Post by Robert Goldman

Post by Attila Lendvai

A cheesy fix would simply be to wrap it in IGNORE-ERRORS. But it
might
cause
errors in its present form.

i've learned, painfully, that indiscriminate ignore-errors will almost
always bite you back (in the form of wasted debugging time), no matter
how innocent they look.

This really is more a QL issue than an ASDF one (although it illustrates
an
issue with wrapping errors, IMO).

while this is true, the implementation of QL requires a reliable way
to hook into the internal state of various versions of ASDF (namely,
into the situation when ASDF is looking for a system, and not finding
it will lead to an error without QL intervention). if i understand it
correctly, this is the crux of this issue.

Sure, and I am happy to try to support this, but not to the extent of
recovering a copy of ASDF 2.x and trying to run it.
The problem is that I don't know when the missing component condition was
added to ASDF, and doing this right would involve checking the enclosed
condition to see if it's a missing component error. I know how to do that
for a modern ASDF, but I don't know how to handle ASDFs that are too old to
have this condition class. And I don't fee like it's my job to think about
that: I think it's perverse to continue trying to use ASDF 2.

ASDF 2.26 is totally unsupported at this point. No implementation uses
anything less than 3.1.2 (the first stable release in the 3.1 series,
from May 2014). Many essential packages require 3.1.x or later. It is
a waste of time to try to get 2.26 running
If Quicklisp wants to be conservative, I would recommend requiring ior
providing ASDF 3.1.7, which is the last in the 3.1 series and pretty
stable, from March 2016, which is two years ago.
The "official" policy of ASDF was always to not support anything that
had already been superseded 2 years ago or more. 3.1.7 should be the
oldest supported ASDF release.
That said, what my opinions seem to be negatively taken into account
by Xach, so there.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics•
http://fare.tunes.org
Evolution competitively selects stable cooperative patterns.