Relativising FASL Cache Paths

[Nicolas Hafner]

Hello everyone.

I have a rather particular problem that I have so far not been able to
solve on my own without brittle workarounds. Specifically, I'm looking
for a way to ensure that the directory paths stored in the FASL cache
directory are, if possible, truncated according to some other path (by
enough-pathname or a similar facility).

The reason I want this is for Portacle <https://portacle.github.io/>.
The Portacle directory should be relocatable and you should even be able
to take it with you on a flash drive to run on other machines. However,
this usually means that the absolute position of the directory on the
file system will change, even if the Lisp files and FASLs within it
remain at the same relative locations.

I would thus like to stop ASDF from recompiling everything when the
directory is moved absolutely, and instead rely on paths relative to
Portacle's own "root" directory if possible.

My current approach involves setting
asdf/output-translations::*output-translation-function* to a modified
version that attempts to relativise the path as seen here
<https://github.com/portacle/config/blob/master/sbcl-init.lisp#L70>.
However, this sometimes leads to issues
<https://github.com/portacle/portacle/issues/62> as the translated path
might be relative and other components down the line seem to expect
absolute paths.

Any advice on how to proceed on this would be appreciated.

Sincerely, Nicolas Hafner

[Faré]

The design of ASDF is that you should properly initialize the
output-translations. The usual way is to use
~/.config/common-lisp/asdf-output-translations.conf, but since in your
case you support the directory moving from one instantiation to the
next, it is probably better to call
asdf:initialize-output-translations at startup with a proper argument
(using properly computed absolute pathnames), just after you (require
"asdf") (which I assume is possible and yielding no older than 3.1.2)
but before you compile anything with it. Untested template that might
work:

(asdf:initialize-output-translations
`(:output-translations
(,(uiop:wilden *portacle-directory*) ,(uiop:wilden
(uiop:subpathname *cache-top* "portacle/")))
(t (uiop:wilden (uiop:subpathname *cache-top* "other/")))
:ignore-inherited-configuration))

Note that you may choose to use the :inherit-configuration instead.

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org
If soldiers are not to cross international boundaries, goods must do so. Unless
the Shackles can be dropped from trade, bombs will be dropped from the sky.
— Otto T. Mallery

Post by Nicolas Hafner
Hello everyone.
I have a rather particular problem that I have so far not been able to solve
on my own without brittle workarounds. Specifically, I'm looking for a way
to ensure that the directory paths stored in the FASL cache directory are,
if possible, truncated according to some other path (by enough-pathname or a
similar facility).
The reason I want this is for Portacle. The Portacle directory should be
relocatable and you should even be able to take it with you on a flash drive
to run on other machines. However, this usually means that the absolute
position of the directory on the file system will change, even if the Lisp
files and FASLs within it remain at the same relative locations.
I would thus like to stop ASDF from recompiling everything when the
directory is moved absolutely, and instead rely on paths relative to
Portacle's own "root" directory if possible.
My current approach involves setting
asdf/output-translations::*output-translation-function* to a modified
version that attempts to relativise the path as seen here. However, this
sometimes leads to issues as the translated path might be relative and other
components down the line seem to expect absolute paths.
Any advice on how to proceed on this would be appreciated.
Sincerely, Nicolas Hafner

[Nicolas Hafner]

I see, thanks a lot for the hint.

I've just stumbled upon another complicating factor though, which is
source location information. Apparently it is not possible to manually
relocate source information after it has been loaded from a FASL in
SBCL. However, if the Lisp files were compiled using a logical pathname,
logical pathname translations could be used to fix that up as required.

Is there a way to transform the input/output files to logical pathnames
to facilitate this, or would that be too much to ask for?

Post by Faré
The design of ASDF is that you should properly initialize the
output-translations. The usual way is to use
~/.config/common-lisp/asdf-output-translations.conf, but since in your
case you support the directory moving from one instantiation to the
next, it is probably better to call
asdf:initialize-output-translations at startup with a proper argument
(using properly computed absolute pathnames), just after you (require
"asdf") (which I assume is possible and yielding no older than 3.1.2)
but before you compile anything with it. Untested template that might
(asdf:initialize-output-translations
`(:output-translations
(,(uiop:wilden *portacle-directory*) ,(uiop:wilden
(uiop:subpathname *cache-top* "portacle/")))
(t (uiop:wilden (uiop:subpathname *cache-top* "other/")))
:ignore-inherited-configuration))
Note that you may choose to use the :inherit-configuration instead.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org
If soldiers are not to cross international boundaries, goods must do so. Unless
the Shackles can be dropped from trade, bombs will be dropped from the sky.
— Otto T. Mallery

Post by Nicolas Hafner
Hello everyone.
I have a rather particular problem that I have so far not been able to solve
on my own without brittle workarounds. Specifically, I'm looking for a way
to ensure that the directory paths stored in the FASL cache directory are,
if possible, truncated according to some other path (by enough-pathname or a
similar facility).
The reason I want this is for Portacle. The Portacle directory should be
relocatable and you should even be able to take it with you on a flash drive
to run on other machines. However, this usually means that the absolute
position of the directory on the file system will change, even if the Lisp
files and FASLs within it remain at the same relative locations.
I would thus like to stop ASDF from recompiling everything when the
directory is moved absolutely, and instead rely on paths relative to
Portacle's own "root" directory if possible.
My current approach involves setting
asdf/output-translations::*output-translation-function* to a modified
version that attempts to relativise the path as seen here. However, this
sometimes leads to issues as the translated path might be relative and other
components down the line seem to expect absolute paths.
Any advice on how to proceed on this would be appreciated.
Sincerely, Nicolas Hafner

[Faré]

Source location via logical pathnames works great on CCL. However, on SBCL,
they run afoul of SBCL's strict enforcement of the standard limitations to
logical pathnames, and you will have quite a hard time with them if you
succeed at all.

One easy but fragile and ugly way to make it work with SBCL would be to
fake logical pathnames using a fixed symlink in $TMPDIR.

More correct solutions that require more hacking include: adding an extra
argument to compile-file by hacking both SBCL and ASDF, to let you override
the default source location stored in a file; or maybe if you're hacking
SBCL, add a non standard logical pathname option. Etc.

My recommendation would be to strive to upstream your patch to whichever
software you hack, and to that end, to speak with the maintainers about
what they're willing to commit.

-#f

Post by Nicolas Hafner
I see, thanks a lot for the hint.
I've just stumbled upon another complicating factor though, which is
source location information. Apparently it is not possible to manually
relocate source information after it has been loaded from a FASL in SBCL.
However, if the Lisp files were compiled using a logical pathname, logical
pathname translations could be used to fix that up as required.
Is there a way to transform the input/output files to logical pathnames to
facilitate this, or would that be too much to ask for?
The design of ASDF is that you should properly initialize the
output-translations. The usual way is to use
~/.config/common-lisp/asdf-output-translations.conf, but since in your
case you support the directory moving from one instantiation to the
next, it is probably better to call
asdf:initialize-output-translations at startup with a proper argument
(using properly computed absolute pathnames), just after you (require
"asdf") (which I assume is possible and yielding no older than 3.1.2)
but before you compile anything with it. Untested template that might
(asdf:initialize-output-translations
`(:output-translations
(,(uiop:wilden *portacle-directory*) ,(uiop:wilden
(uiop:subpathname *cache-top* "portacle/")))
(t (uiop:wilden (uiop:subpathname *cache-top* "other/")))
:ignore-inherited-configuration))
Note that you may choose to use the :inherit-configuration instead.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org
If soldiers are not to cross international boundaries, goods must do so. Unless
the Shackles can be dropped from trade, bombs will be dropped from the sky.
— Otto T. Mallery
Hello everyone.
I have a rather particular problem that I have so far not been able to solve
on my own without brittle workarounds. Specifically, I'm looking for a way
to ensure that the directory paths stored in the FASL cache directory are,
if possible, truncated according to some other path (by enough-pathname or a
similar facility).
The reason I want this is for Portacle. The Portacle directory should be
relocatable and you should even be able to take it with you on a flash drive
to run on other machines. However, this usually means that the absolute
position of the directory on the file system will change, even if the Lisp
files and FASLs within it remain at the same relative locations.
I would thus like to stop ASDF from recompiling everything when the
directory is moved absolutely, and instead rely on paths relative to
Portacle's own "root" directory if possible.
My current approach involves setting
asdf/output-translations::*output-translation-function* to a modified
version that attempts to relativise the path as seen here. However, this
sometimes leads to issues as the translated path might be relative and other
components down the line seem to expect absolute paths.
Any advice on how to proceed on this would be appreciated.
Sincerely, Nicolas Hafner

[73budden .]

Hi!

Nicolas, can you resolve source locations at SWANK level? E.g.
functions like SWANK/SBCL::SOURCE-FILE-SOURCE-LOCATION might be
appropriate for patching: you take the location that SBCL gives you,
but then redirect SWANK to other place. I guess this is not the only
place you have to patch, it would require patching private functions
and it might not work at all. But if that works for you please let me
know. I'm also maintaining a thing a bit similar to portacle and it
looks like I would suffer from the same issue. Also let me remind you
that you can set up the path to SBCL sources by

sb-ext:set-sbcl-source-location

AFAIR locating sources of SBCL functions didn't work in portacle when
I saw it, and it was not very long ago.

[Nicolas Hafner]

Post by 73budden .
Nicolas, can you resolve source locations at SWANK level? E.g.
functions like SWANK/SBCL::SOURCE-FILE-SOURCE-LOCATION might be
appropriate for patching: you take the location that SBCL gives you,
but then redirect SWANK to other place. I guess this is not the only
place you have to patch, it would require patching private functions
and it might not work at all. But if that works for you please let me
know. I'm also maintaining a thing a bit similar to portacle and it
looks like I would suffer from the same issue.

That sounds like a really nasty workaround. I'd much rather try to get
ASDF to use logical pathnames.

Post by 73budden .
Also let me remind you that you can set up the path to SBCL sources by
sb-ext:set-sbcl-source-location
AFAIR locating sources of SBCL functions didn't work in portacle when
I saw it, and it was not very long ago.

Portacle already does that
<https://github.com/portacle/config/blob/master/sbcl-init.lisp#L52>.

[Faré]

Post by 73budden .
Nicolas, can you resolve source locations at SWANK level? E.g.
functions like SWANK/SBCL::SOURCE-FILE-SOURCE-LOCATION might be
appropriate for patching: you take the location that SBCL gives you,
but then redirect SWANK to other place. I guess this is not the only
place you have to patch, it would require patching private functions
and it might not work at all. But if that works for you please let me
know. I'm also maintaining a thing a bit similar to portacle and it
looks like I would suffer from the same issue.
That sounds like a really nasty workaround. I'd much rather try to get
ASDF to use logical pathnames.

Logical pathnames are never the solution, especially not if portability
means anything, and if SBCL is targeted in particular.

Too many systems have file names that just can't be valid logical pathnames
(that are limited strictly to single-case letters, digits, and dashes, plus
a dot only to separate the similarly-limited type). Underscores, mixed
case, wrong case, plus sign, dot, and whatever else are not allowed, yet
may be implied by the .asd file.

Good luck convincing all authors in Quicklisp of renaming their files and
directories (though this renaming could conceivably be done automatically
after the fact, combined with large logical pathname mapping tables, for
atrocious performance and horrible bugs when that fails; good luck with
that kluge).

I recommend aiming for a true solution, which implies adding the capability
of overriding the pathname to SBCL's compile-file then UIOP's compile-file*
and ASDF's compile-op, and/or non-compliant lax pass-through logical
pathnames a la CCL.

-#f