How accurate is FILELIST.TXT ?

[John Forkosh]

First, please see
https://unix.stackexchange.com/questions/538188/
for background (subject: rsync-like-delete-functionality-with-wget).

Briefly, on 20-March-2019 I mirrored -current using
wget -r -np -R "index.html*"
https://mirror.slackbuilds.org/slackware/slackware64-current/
and recently, on 29-Aug, I updated/refreshed my mirror by
adding the -N option to the otherwise-identical wget command.

So that worked, except that my updated mirror now contains both
older and newer versions of many packages, e.g.,
SDL2_mixer-2.0.4-x86_64-1.txz
SDL2_mixer-2.0.4-x86_64-2.txz
libcddb-1.3.2-x86_64-5.txz
libcddb-1.3.2-x86_64-6.txz
and lots, lots more.
So how to --delete the old ones??? That's the overall problem
I'm trying to solve.

The stackexchange post contains a tentative solution by @roaima,
which I'm in the middle of testing. But it wasn't quite working
right. So I went back to my original, un-updated 20-March mirror,
and tried the procedure on it, figuring the FILELIST.TXT there
should exactly match the actual files comprising the mirror.

Guess what (you already guessed, didn't you?:). As far as I can
tell, there are 345 files in the 20-March FILELIST.TXT that aren't
in the 20-March mirror. The very first few are
./source/a/eudev/config/modprobe.d/8139cp.conf
./source/a/eudev/config/modprobe.d/README
./source/a/eudev/config/modprobe.d/bcm43xx.conf
./source/a/eudev/config/modprobe.d/edac.conf
./source/a/eudev/config/modprobe.d/eepro100.conf
./source/a/eudev/config/modprobe.d/eth1394.conf
./source/a/eudev/config/modprobe.d/evbug.conf
Indeed, my mirror's ./source/a/eudev/config/ directory contains
several subdirectories, all of which are entirely empty.
But FILELIST.TXT contains files in each and every one of those
subdirectories. Moreover, conversely, my mirror contains (again,
that's "as far as I can tell" so far) 64 files not in FILELIST.TXT.

So, @roaima's suggestion is kind of moot without an accurate
FILELIST.TXT to begin with. Is there any way to get one,
or am I wrong about it being wrong, or what?

And, even better, as per the original stackexchange question,
is there a more straightforward way of updating my slackbuilds.org
mirror, and --delete'ing any old files/package-versions from
my mirror that no longer exist in -current? Thanks,

[dillinger]

Post by John Forkosh
First, please see
https://unix.stackexchange.com/questions/538188/
for background (subject: rsync-like-delete-functionality-with-wget).

To me it looks like you don't need wget, you need a mirror you can rsync
with, try something like this:

#!/bin/bash
#
# rsync slackware64-current
#

SOURCE="ftp.nluug.nl::slackware/slackware64-current/"

DEST="/mnt/data/slackware/slackware64-current/"

OPTIONS="-avu --delete-excluded"

# Save some space & bandwidth
EXCLUDE="--include=extra/source/ --exclude=source/ --exclude=kde/
--exclude=kdei/ --exclude=t/ --exclude=aspell-word-lists/"

rsync $OPTIONS $EXCLUDE $SOURCE $DEST
# --dry-run

[John Forkosh]

Post by dillinger

Post by John Forkosh
First, please see
https://unix.stackexchange.com/questions/538188/
for background (subject: rsync-like-delete-functionality-with-wget).

To me it looks like you don't need wget,
you need a mirror you can rsync with,

Indeed!!! That's exactly what the stackexchange post says,
i.e., that I wanted to use rsync, but don't have any kind
of account on slackbuilds, whereby (as per that post)
rsync -av --delete ***@slackbuilds...etc
would be precisely what I want to accomplish in one easy step.

Post by dillinger
#!/bin/bash
#
# rsync slackware64-current
#
SOURCE="ftp.nluug.nl::slackware/slackware64-current/"
DEST="/mnt/data/slackware/slackware64-current/"
OPTIONS="-avu --delete-excluded"
# Save some space & bandwidth
EXCLUDE="--include=extra/source/ --exclude=source/ --exclude=kde/
--exclude=kdei/ --exclude=t/ --exclude=aspell-word-lists/"
rsync $OPTIONS $EXCLUDE $SOURCE $DEST
# --dry-run

Thanks, Dillinger. So I take it that
ftp.nluug.nl::slackware/slackware64-current/
is a mirror that'll let me rsync from it without
a ***@... "prefix". Terrific!!!

I just tried
rsync -av --delete -n ftp.nluug.nl::slackware/slackware64-current/ \
slackware64-current-29Aug2019/ > junk
where that target contains my recent mirror using wget.
Seems to be deleting 1748 files, but also actually updating 1593.
Oh, wait, most of those seem to be directories, where it's probably
just the timestamp that needs to be updated. I'll have to check
everything more carefully, but looks like this might be the answer
I was hoping for. Thanks, again.

[John Forkosh]

<<snip>>

Post by John Forkosh

Post by dillinger
#!/bin/bash
#
# rsync slackware64-current
#
SOURCE="ftp.nluug.nl::slackware/slackware64-current/"
DEST="/mnt/data/slackware/slackware64-current/"
OPTIONS="-avu --delete-excluded"
# Save some space & bandwidth
EXCLUDE="--include=extra/source/ --exclude=source/ --exclude=kde/
--exclude=kdei/ --exclude=t/ --exclude=aspell-word-lists/"
rsync $OPTIONS $EXCLUDE $SOURCE $DEST
# --dry-run

Thanks, Dillinger. So I take it that
ftp.nluug.nl::slackware/slackware64-current/
is a mirror that'll let me rsync from it without
I just tried
rsync -av --delete -n ftp.nluug.nl::slackware/slackware64-current/ \
slackware64-current-29Aug2019/ > junk
where that target contains my recent mirror using wget.
Seems to be deleting 1748 files, but also actually updating 1593.
Oh, wait, most of those seem to be directories, where it's probably
just the timestamp that needs to be updated. I'll have to check
everything more carefully, but looks like this might be the answer
I was hoping for. Thanks, again.

In addition to your rsync site
ftp.nluug.nl::slackware/slackware64-current/
I tried the site (from list at https://mirrors.slackware.com/mirrorlist/)
rsync://mirror.math.princeton.edu/pub/slackware/slackware64-current/
And both do the exactly identical same thing (the rsync outputs diff
identically after sort'ing). So I'm assuming they're both doing the
same right thing, and applied -av --delete to (a copy of) my mirror.
By the way, that "updated 1593" above is due not only to the
directory timestamps, but also because I'd originally forgot that
wget's recursion default is five levels down. And that left me with
a few empty directories that rsync now filled in. But those were
mostly in source/ which didn't affect the usb stick created by
usb-and-pxe-installers/usbimg2disk.sh, so I never noticed the
problem until now. Thanks again.

[Rich]

Post by John Forkosh
First, please see
https://unix.stackexchange.com/questions/538188/
for background (subject: rsync-like-delete-functionality-with-wget).
Briefly, on 20-March-2019 I mirrored -current using
wget -r -np -R "index.html*"
https://mirror.slackbuilds.org/slackware/slackware64-current/

No, your wget command line above mirrored Slackbuilds and got
Slackbuilds set of -current scripts (which don't exist). Which makes
the rest of your post quite a bit suspect for accuracy.

Post by John Forkosh
and recently, on 29-Aug, I updated/refreshed my mirror by
adding the -N option to the otherwise-identical wget command.
So that worked, except that my updated mirror now contains both
older and newer versions of many packages, e.g.,
...
So how to --delete the old ones??? That's the overall problem
I'm trying to solve.

Use rsync....

See the 'repository' pages on slackbuilds:
https://slackbuilds.org/repository/14.2/ at the bottom of which are
these links:

Access to the repository is available via:
ftp git cgit http *rsync*

And if you want to sync the Slackbuilds repo, use the 'rsync' access.
Then rsync will handle the deletion you want.

[John Forkosh]

Post by Rich

Post by John Forkosh
First, please see
https://unix.stackexchange.com/questions/538188/
for background (subject: rsync-like-delete-functionality-with-wget).
Briefly, on 20-March-2019 I mirrored -current using
wget -r -np -R "index.html*"
https://mirror.slackbuilds.org/slackware/slackware64-current/

No, your wget command line above mirrored Slackbuilds and got
Slackbuilds set of -current scripts (which don't exist).

No (to your "No"), I ran exactly that wget command and got a mirror.
And from that mirror I ran the script
usb-and-pxe-installers/usbimg2disk.sh
which created a usb stick with the istall image and all software.
And I booted from that stick, and did a "full" install, which
worked as usual. And I'm running that install right now. No problems.

Post by Rich
Which makes the rest of your post quite a bit suspect for accuracy.

I pretty much just reported what I did and what happened,
and then asked for help to accomplish what I'd wanted to happen.
So what I did may have been the wrong thing to do, but
the reported facts are accurate (at least to the best of my
recollection and notes).

Post by Rich

Post by John Forkosh
and recently, on 29-Aug, I updated/refreshed my mirror by
adding the -N option to the otherwise-identical wget command.
So that worked, except that my updated mirror now contains both
older and newer versions of many packages, e.g.,
...
So how to --delete the old ones??? That's the overall problem
I'm trying to solve.

Use rsync....
https://slackbuilds.org/repository/14.2/ at the bottom of which are
ftp git cgit http *rsync*
And if you want to sync the Slackbuilds repo, use the 'rsync' access.
Then rsync will handle the deletion you want.

Yup, rsync's exactly what I wanted to use, as per that stackexhange post
(and as per my preceding followup to dillinger). Thanks for the link
to a site that'll allow rsync access without an account. That should
completely solve the problem.

[John Forkosh]

<<snip>>

Post by John Forkosh

Post by Rich
Use rsync....
https://slackbuilds.org/repository/14.2/ at the bottom of which are
ftp git cgit http *rsync*
And if you want to sync the Slackbuilds repo, use the 'rsync' access.
Then rsync will handle the deletion you want.

Yup, rsync's exactly what I wanted to use, as per that stackexhange post
(and as per my preceding followup to dillinger). Thanks for the link
to a site that'll allow rsync access without an account. That should
completely solve the problem.

Oops, that rsync link is to slackbuilds.org/slackbuilds/ which only
contains 14.2, not -current/. Now, there is a -current at
mirror.slackbuilds.org/slackware/ but that didn't allow me rsync access.
And there's no corresponding slackbuilds.org/slackware/ directory at all.
So dillinger's suggestion of ftp.nluug.nl::slackware/slackware64-current/
seems to be the only -current/ which I'm aware of that allows rsync.
But I'd imagine there are likely others. Is there a list of
rsync-accessible sites containing -current/ ?

[John Forkosh]

Is there a list of rsync-accessible sites containing -current/ ?

(Sorry for following myself up a second time, but...)
google pointed me to the list at
https://mirrors.slackware.com/mirrorlist/
which has a bunch of rsync-accessible mirrors at the bottom of the page.
Not sure which ones do/don't have -current/, but I imagine at least
several do.