FYI: WITH_REPRODUCIBLE_BUILD= problem for some files?

Discussion:

Mark Millard via freebsd-current

2021-04-29 06:50:33 UTC

I did 2 test buildworld's based on:

# ~/fbsd-based-on-what-freebsd.sh
branch: releng/13.0
merge-base: ea31abc261ffc01b6ff5671bffb15cf910a07f4b
merge-base: CommitDate: 2021-04-09 00:14:30 +0000
ea31abc261ff (HEAD -> releng/13.0, tag: release/13.0.0, freebsd/releng/13.0) 13.0: update to RELEASE
n244733 (--first-parent --count for merge-base)

and produced separate build trees. I also installed the
world build into two separate directory trees:

/usr/obj/DESTDIRs/13_0R-CA7-chroot/
vs.
/usr/obj/DESTDIRs/13_0R-CA7-poud/

This was for other reasons. But eventually I happened
to do a diff -rq of the two trees and ended up with
the output showing some differing files:

Files /usr/obj/DESTDIRs/13_0R-CA7-chroot/sbin/ping and /usr/obj/DESTDIRs/13_0R-CA7-poud/sbin/ping differ
Files /usr/obj/DESTDIRs/13_0R-CA7-chroot/sbin/ping6 and /usr/obj/DESTDIRs/13_0R-CA7-poud/sbin/ping6 differ
Files /usr/obj/DESTDIRs/13_0R-CA7-chroot/usr/bin/ntpq and /usr/obj/DESTDIRs/13_0R-CA7-poud/usr/bin/ntpq differ
Files /usr/obj/DESTDIRs/13_0R-CA7-chroot/usr/lib/debug/sbin/ping.debug and /usr/obj/DESTDIRs/13_0R-CA7-poud/usr/lib/debug/sbin/ping.debug differ
Files /usr/obj/DESTDIRs/13_0R-CA7-chroot/usr/lib/debug/usr/sbin/ntpd.debug and /usr/obj/DESTDIRs/13_0R-CA7-poud/usr/lib/debug/usr/sbin/ntpd.debug differ
Files /usr/obj/DESTDIRs/13_0R-CA7-chroot/usr/lib/debug/usr/tests/sbin/ping/in_cksum_test.debug and /usr/obj/DESTDIRs/13_0R-CA7-poud/usr/lib/debug/usr/tests/sbin/ping/in_cksum_test.debug differ
Files /usr/obj/DESTDIRs/13_0R-CA7-chroot/usr/sbin/ntp-keygen and /usr/obj/DESTDIRs/13_0R-CA7-poud/usr/sbin/ntp-keygen differ
Files /usr/obj/DESTDIRs/13_0R-CA7-chroot/usr/sbin/ntpd and /usr/obj/DESTDIRs/13_0R-CA7-poud/usr/sbin/ntpd differ
Files /usr/obj/DESTDIRs/13_0R-CA7-chroot/usr/sbin/ntpdate and /usr/obj/DESTDIRs/13_0R-CA7-poud/usr/sbin/ntpdate differ
Files /usr/obj/DESTDIRs/13_0R-CA7-chroot/usr/sbin/ntpdc and /usr/obj/DESTDIRs/13_0R-CA7-poud/usr/sbin/ntpdc differ
Files /usr/obj/DESTDIRs/13_0R-CA7-chroot/usr/sbin/sntp and /usr/obj/DESTDIRs/13_0R-CA7-poud/usr/sbin/sntp differ
Files /usr/obj/DESTDIRs/13_0R-CA7-chroot/usr/tests/sbin/ping/in_cksum_test and /usr/obj/DESTDIRs/13_0R-CA7-poud/usr/tests/sbin/ping/in_cksum_test differ

(That is all.)

For as much as I've looked at (not much), it looks to be
variations in byte-padding values.

The builds both were set up to tune for cortex-a7 explicitly.
I patch top's source code. I patch the OOM kill code to report
the specific reason for a kill. I still have some bcm2838
pci/xhci patching in place from an old investigation, but
that would be kernel code. None of the patching is specific
to the above list of files.

The hosting context was:

# uname -apKU
FreeBSD CA72_4c8G_ZFS 13.0-RELEASE FreeBSD 13.0-RELEASE #1 releng/13.0-n244733-ea31abc261ff-dirty: Wed Apr 28 05:45:27 PDT 2021 ***@CA72_4c8G_ZFS:/usr/obj/BUILDs/13_0R-CA72-nodbg-clang/usr/src/arm64.aarch64/sys/GENERIC-NODBG-CA72 arm64 aarch64 1300139 1300139

based on building the same source code (tuning for
cortex-a72).

It was the same media for all the activity. Unlike
the past many years for me, the context is using ZFS
instead of UFS, not that I think that makes a
difference here.

The differences do not mess up my activity but others
might notice and care about such differences.

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

Ed Maste

2021-05-03 14:47:02 UTC

Permalink

On Thu, 29 Apr 2021 at 02:50, Mark Millard via freebsd-current

Post by Mark Millard via freebsd-current
Files /usr/obj/DESTDIRs/13_0R-CA7-chroot/sbin/ping and /usr/obj/DESTDIRs/13_0R-CA7-poud/sbin/ping differ
Files /usr/obj/DESTDIRs/13_0R-CA7-chroot/sbin/ping6 and /usr/obj/DESTDIRs/13_0R-CA7-poud/sbin/ping6 differ
Files /usr/obj/DESTDIRs/13_0R-CA7-chroot/usr/bin/ntpq and /usr/obj/DESTDIRs/13_0R-CA7-poud/usr/bin/ntpq differ...

This is unexpected. Unfortunately I haven't looked at reproducibility
in a while, and my work was all on x86. This could be a regression or
a longstanding issue with arm64.

If you install the diffoscope package (py37-diffoscope) and run it on
the two directories / files it should give a more convenient view of
the differences. (Or, if you can make a tarball of the differing files
I can take a look.)

Mark Millard via freebsd-current

2021-05-03 17:51:55 UTC

Permalink

Post by Ed Maste
On Thu, 29 Apr 2021 at 02:50, Mark Millard via freebsd-current

This is unexpected. Unfortunately I haven't looked at reproducibility
in a while, and my work was all on x86. This could be a regression or
a longstanding issue with arm64.
If you install the diffoscope package (py37-diffoscope) and run it on
the two directories / files it should give a more convenient view of
the differences. (Or, if you can make a tarball of the differing files
I can take a look.)

I no longer have the same content in those directory
trees: newer rebuild and the same buildworld used to
installworld to both places, instead of 2 different
buildworld's. I'm also unsure how reproducible getting
differences was.

I can eventually do experiments to test multiple separate
buildworld's and installworld's, but the machine is busy
building ports and the llvm builds involved means it
will be some time before I'd switch activities. And the
buildworld's involve llvm builds as well and take notable
time themselves. So my next comparison will not be any
time soon.

I'll let you know if I manage to generate another example,
this time being sure to keep the data. If I try multiple
times without finding any differences, I'll eventually
decide "enough is enough" and let you know.

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

Mark Millard via freebsd-current

2021-05-04 02:26:04 UTC

Permalink

Post by Mark Millard via freebsd-current

Post by Ed Maste
On Thu, 29 Apr 2021 at 02:50, Mark Millard via freebsd-current

This is unexpected. Unfortunately I haven't looked at reproducibility
in a while, and my work was all on x86. This could be a regression or
a longstanding issue with arm64.
If you install the diffoscope package (py37-diffoscope) and run it on
the two directories / files it should give a more convenient view of
the differences. (Or, if you can make a tarball of the differing files
I can take a look.)

I no longer have the same content in those directory
trees: newer rebuild and the same buildworld used to
installworld to both places, instead of 2 different
buildworld's. I'm also unsure how reproducible getting
differences was.
I can eventually do experiments to test multiple separate
buildworld's and installworld's, but the machine is busy
building ports and the llvm builds involved means it
will be some time before I'd switch activities. And the
buildworld's involve llvm builds as well and take notable
time themselves. So my next comparison will not be any
time soon.
I'll let you know if I manage to generate another example,
this time being sure to keep the data. If I try multiple
times without finding any differences, I'll eventually
decide "enough is enough" and let you know.

I've still got a long ways to go to do the first
actual comparison of builds.

But I'll note that I've built and stalled py37-diffoscope
(new to me). A basic quick test showed that it reports:

W: diffoscope.main: Fuzzy-matching is currently disabled as the "tlsh" module is unavailable.

As I'm not familiar with the tool, you might need to send
notes about how you want me to use the tool to get the
output that you would want. (And, so, I get to learn . . .)

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

Mark Millard via freebsd-current

2021-05-04 04:27:07 UTC

Permalink

Post by Mark Millard via freebsd-current

Post by Ed Maste
On Thu, 29 Apr 2021 at 02:50, Mark Millard via freebsd-current

This is unexpected. Unfortunately I haven't looked at reproducibility
in a while, and my work was all on x86. This could be a regression or
a longstanding issue with arm64.
If you install the diffoscope package (py37-diffoscope) and run it on
the two directories / files it should give a more convenient view of
the differences. (Or, if you can make a tarball of the differing files
I can take a look.)

I no longer have the same content in those directory
trees: newer rebuild and the same buildworld used to
installworld to both places, instead of 2 different
buildworld's. I'm also unsure how reproducible getting
differences was.
I can eventually do experiments to test multiple separate
buildworld's and installworld's, but the machine is busy
building ports and the llvm builds involved means it
will be some time before I'd switch activities. And the
buildworld's involve llvm builds as well and take notable
time themselves. So my next comparison will not be any
time soon.
I'll let you know if I manage to generate another example,
this time being sure to keep the data. If I try multiple
times without finding any differences, I'll eventually
decide "enough is enough" and let you know.

I've still got a long ways to go to do the first
actual comparison of builds.
But I'll note that I've built and stalled py37-diffoscope
W: diffoscope.main: Fuzzy-matching is currently disabled as the "tlsh" module is unavailable.
As I'm not familiar with the tool, you might need to send
notes about how you want me to use the tool to get the
output that you would want. (And, so, I get to learn . . .)

I've tried another experiment (* in the path matches "28" and "30"):

# diffoscope /.zfs/snapshot/2021-04-*-01:40:48-0/bin/sh
$<3/>2021-05-03 21:08:48 W: diffoscope.main: Fuzzy-matching is currently disabled as the "tlsh" module is unavailable.
$<3/>Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/diffoscope/main.py", line 745, in main
sys.exit(run_diffoscope(parsed_args))
File "/usr/local/lib/python3.7/site-packages/diffoscope/main.py", line 677, in run_diffoscope
difference = load_diff_from_path(path1)
File "/usr/local/lib/python3.7/site-packages/diffoscope/readers/__init__.py", line 31, in load_diff_from_path
return load_diff(codecs.getreader("utf-8")(fp), path)
File "/usr/local/lib/python3.7/site-packages/diffoscope/readers/__init__.py", line 35, in load_diff
return JSONReaderV1().load(fp, path)
File "/usr/local/lib/python3.7/site-packages/diffoscope/readers/json.py", line 33, in load
raw = json.load(fp)
File "/usr/local/lib/python3.7/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/usr/local/lib/python3.7/codecs.py", line 504, in read
newchars, decodedbytes = self.decode(data, self.errors)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb7 in position 18: invalid start byte

The two older snapshots of a Boot Environment have
bin/sh files that compare equal. But every program I
tried the above sort of thing against on got the same
UnicodeDecodeError result from diffoscope, byte value
and position matching.

These snapshots have more than an installworld in them
and so are messy to compare overall. But the
installworld (and installkernel) content show similar
differences to what I reported before as far as
example files with differences go. But this is aarch64,
not armv7.

It will still be notable time before I have simple
installworld tree's to compare.

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

Mark Millard via freebsd-current

2021-05-04 07:10:16 UTC

Permalink

Post by Mark Millard via freebsd-current

Post by Ed Maste
On Thu, 29 Apr 2021 at 02:50, Mark Millard via freebsd-current

This is unexpected. Unfortunately I haven't looked at reproducibility
in a while, and my work was all on x86. This could be a regression or
a longstanding issue with arm64.
If you install the diffoscope package (py37-diffoscope) and run it on
the two directories / files it should give a more convenient view of
the differences. (Or, if you can make a tarball of the differing files
I can take a look.)

I no longer have the same content in those directory
trees: newer rebuild and the same buildworld used to
installworld to both places, instead of 2 different
buildworld's. I'm also unsure how reproducible getting
differences was.
I can eventually do experiments to test multiple separate
buildworld's and installworld's, but the machine is busy
building ports and the llvm builds involved means it
will be some time before I'd switch activities. And the
buildworld's involve llvm builds as well and take notable
time themselves. So my next comparison will not be any
time soon.
I'll let you know if I manage to generate another example,
this time being sure to keep the data. If I try multiple
times without finding any differences, I'll eventually
decide "enough is enough" and let you know.

# diffoscope /.zfs/snapshot/2021-04-*-01:40:48-0/bin/sh
$<3/>2021-05-03 21:08:48 W: diffoscope.main: Fuzzy-matching is currently disabled as the "tlsh" module is unavailable.
File "/usr/local/lib/python3.7/site-packages/diffoscope/main.py", line 745, in main
sys.exit(run_diffoscope(parsed_args))
File "/usr/local/lib/python3.7/site-packages/diffoscope/main.py", line 677, in run_diffoscope
difference = load_diff_from_path(path1)
File "/usr/local/lib/python3.7/site-packages/diffoscope/readers/__init__.py", line 31, in load_diff_from_path
return load_diff(codecs.getreader("utf-8")(fp), path)
File "/usr/local/lib/python3.7/site-packages/diffoscope/readers/__init__.py", line 35, in load_diff
return JSONReaderV1().load(fp, path)
File "/usr/local/lib/python3.7/site-packages/diffoscope/readers/json.py", line 33, in load
raw = json.load(fp)
File "/usr/local/lib/python3.7/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/usr/local/lib/python3.7/codecs.py", line 504, in read
newchars, decodedbytes = self.decode(data, self.errors)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb7 in position 18: invalid start byte
The two older snapshots of a Boot Environment have
bin/sh files that compare equal. But every program I
tried the above sort of thing against on got the same
UnicodeDecodeError result from diffoscope, byte value
and position matching.
These snapshots have more than an installworld in them
and so are messy to compare overall. But the
installworld (and installkernel) content show similar
differences to what I reported before as far as
example files with differences go. But this is aarch64,
not armv7.
It will still be notable time before I have simple
installworld tree's to compare.

While waiting for the 2nd buildworld to complete, I used
the 1st buildworld's materials to installworld twice (to
different, empty directory trees) and then did a diff.
The diff reported:

# diff -rq /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/ /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm2/ | more
diff: /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/usr/tests/local: No such file or directory
diff: /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/usr/tests/sys/pjdfstest/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests: Too many levels of symbolic links

FYI:

# ls -Tld /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/usr/tests/local
lrwxr-xr-x 1 root wheel 14 May 3 23:18:22 2021 /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/usr/tests/local -> ../local/tests

So this looks like the expected result and suggests that
the problem(s) are at the buildworld stage. (No surprise.)

I should be sleeping by the time the 2nd buildworld for
cortex-a72 (aarch64) is done. (That is what I decided
to test initially.) So it will be more hours until I have
basic comparison results from 2 distinct buildworlds.

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

Ed Maste

2021-05-04 13:01:34 UTC

Permalink

Post by Mark Millard via freebsd-current
But I'll note that I've built and stalled py37-diffoscope
W: diffoscope.main: Fuzzy-matching is currently disabled as the "tlsh" module is unavailable.

I just looked up tlsh - its "A Locality Sensitive Hash"; I presume
diffoscope uses it to infer file renames. I believe the warning
emitted here should have no impact on the output we're looking for.

As far as the utf-8 issues go, diffoscope requires a utf-8 locale and
I suspect that is the issue. If you don't have LANG set already, try
setting LANG=C.UTF-8 in your environment.

Mark Millard via freebsd-current

2021-05-04 15:51:50 UTC

Permalink

Post by Ed Maste

Post by Mark Millard via freebsd-current
But I'll note that I've built and stalled py37-diffoscope
W: diffoscope.main: Fuzzy-matching is currently disabled as the "tlsh" module is unavailable.

I just looked up tlsh - its "A Locality Sensitive Hash"; I presume
diffoscope uses it to infer file renames. I believe the warning
emitted here should have no impact on the output we're looking for.

Okay.

Post by Ed Maste
As far as the utf-8 issues go, diffoscope requires a utf-8 locale and
I suspect that is the issue. If you don't have LANG set already, try
setting LANG=C.UTF-8 in your environment.

That is not the issue for the UnicodeDecodeError:

# echo $LANG
C.UTF-8

# diffoscope /.zfs/snapshot/2021-04-*-01:40:48-0/bin/sh
$<3/>2021-05-04 08:49:21 W: diffoscope.main: Fuzzy-matching is currently disabled as the "tlsh" module is unavailable.
$<3/>Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/diffoscope/main.py", line 745, in main
sys.exit(run_diffoscope(parsed_args))
File "/usr/local/lib/python3.7/site-packages/diffoscope/main.py", line 677, in run_diffoscope
difference = load_diff_from_path(path1)
File "/usr/local/lib/python3.7/site-packages/diffoscope/readers/__init__.py", line 31, in load_diff_from_path
return load_diff(codecs.getreader("utf-8")(fp), path)
File "/usr/local/lib/python3.7/site-packages/diffoscope/readers/__init__.py", line 35, in load_diff
return JSONReaderV1().load(fp, path)
File "/usr/local/lib/python3.7/site-packages/diffoscope/readers/json.py", line 33, in load
raw = json.load(fp)
File "/usr/local/lib/python3.7/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/usr/local/lib/python3.7/codecs.py", line 504, in read
newchars, decodedbytes = self.decode(data, self.errors)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb7 in position 18: invalid start byte

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

Ed Maste

2021-05-04 16:55:52 UTC

Permalink

Post by Mark Millard via freebsd-current

Post by Ed Maste
As far as the utf-8 issues go, diffoscope requires a utf-8 locale and
I suspect that is the issue. If you don't have LANG set already, try
setting LANG=C.UTF-8 in your environment.

# echo $LANG
C.UTF-8
# diffoscope /.zfs/snapshot/2021-04-*-01:40:48-0/bin/sh
[...]
$<3/>2021-05-04 08:49:21 W: diffoscope.main: Fuzzy-matching is currently disabled as the "tlsh" module is unavailable.
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb7 in position 18: invalid start byte

Hmm, interesting - if you don't mind sharing I'd be interested in a
copy of /.zfs/snapshot/2021-04-*-01:40:48-0/bin/sh, in order to track
down what appears to be a diffoscope issue.

To investigate the non-reproducibility though we can just manually run
through the same sort of process that Diffoscope uses. I would suggest
cmp -x <file1> <file2> to find the offsets of the difference(s), then
use readelf -S <file> to determine which section(s) have differences.

Mark Millard via freebsd-current

2021-05-04 17:01:15 UTC

Permalink

This post might be inappropriate. Click to display it.

Mark Millard via freebsd-current

2021-05-04 17:17:27 UTC

Permalink

[Just adding readelf -S info since it seems to show more.]

Post by Mark Millard via freebsd-current

Post by Ed Maste

Post by Mark Millard via freebsd-current
But I'll note that I've built and stalled py37-diffoscope
W: diffoscope.main: Fuzzy-matching is currently disabled as the "tlsh" module is unavailable.

I just looked up tlsh - its "A Locality Sensitive Hash"; I presume
diffoscope uses it to infer file renames. I believe the warning
emitted here should have no impact on the output we're looking for.

Okay.

Post by Ed Maste
As far as the utf-8 issues go, diffoscope requires a utf-8 locale and
I suspect that is the issue. If you don't have LANG set already, try
setting LANG=C.UTF-8 in your environment.

# echo $LANG
C.UTF-8
# diffoscope /.zfs/snapshot/2021-04-*-01:40:48-0/bin/sh
$<3/>2021-05-04 08:49:21 W: diffoscope.main: Fuzzy-matching is currently disabled as the "tlsh" module is unavailable.
File "/usr/local/lib/python3.7/site-packages/diffoscope/main.py", line 745, in main
sys.exit(run_diffoscope(parsed_args))
File "/usr/local/lib/python3.7/site-packages/diffoscope/main.py", line 677, in run_diffoscope
difference = load_diff_from_path(path1)
File "/usr/local/lib/python3.7/site-packages/diffoscope/readers/__init__.py", line 31, in load_diff_from_path
return load_diff(codecs.getreader("utf-8")(fp), path)
File "/usr/local/lib/python3.7/site-packages/diffoscope/readers/__init__.py", line 35, in load_diff
return JSONReaderV1().load(fp, path)
File "/usr/local/lib/python3.7/site-packages/diffoscope/readers/json.py", line 33, in load
raw = json.load(fp)
File "/usr/local/lib/python3.7/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/usr/local/lib/python3.7/codecs.py", line 504, in read
newchars, decodedbytes = self.decode(data, self.errors)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb7 in position 18: invalid start byte

Well, the list of differing files is huge. But this seems to
be .gnu_debuglink content for the area it is in.

Specifically: the last 4 bytes of the .gnu_debuglink section.

Post by Mark Millard via freebsd-current
I'll note
that I did installworld but not the likes of distrib-dirs
or distribution this time.
zroot/BUILDs/13_0R-CA72-nodbg-clang 5.13G 118G 5.13G /usr/obj/BUILDs/13_0R-CA72-nodbg-clang
zroot/BUILDs/13_0R-CA72-nodbg-clang-alt 4.28G 118G 4.28G /usr/obj/BUILDs/13_0R-CA72-nodbg-clang-alt
zroot/DESTDIRs/13_0R-CA72-instwrld-alt 1.44G 118G 1.44G /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt
zroot/DESTDIRs/13_0R-CA72-instwrld-norm 1.44G 118G 1.44G /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm
Previously (armv7 target) I had built, installed, rebuilt
to same directory (after clean-out) and installed to an
alternate directory. That had gotten only a few files
different but I do not know (yet) if it was the procedural
difference that made the difference.
# diff -rq /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/ /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/ | more
Files /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/bin/[ and /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/bin/[ differ
Files /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/bin/cat and /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/bin/cat differ
Files /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/bin/chflags and /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/bin/chflags differ
Files /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/bin/chio and /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/bin/chio differ
. . .
Looking, aarch64 seems to typically get a back-to-back
sequence of 4 bytes different in native programs in my
# cmp -x /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/bin/cat /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/bin/cat
00003bd4 1d 65
00003bd5 eb a3
00003bd6 bb ca
00003bd7 8e 1a
# ls -Tld /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/bin/cat /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/bin/cat
-r-xr-xr-x 1 root wheel 18448 May 4 08:55:01 2021 /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/bin/cat
-r-xr-xr-x 1 root wheel 18448 May 3 23:16:36 2021 /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/bin/cat
Idx Name Size VMA LMA File off Algn
. . .
25 .gnu_debuglink 00000010 0000000000000000 0000000000000000 00003bc8 2**0
CONTENTS, READONLY

Section Headers:
[Nr] Name Type Address Offset
Size EntSize Flags Link Info Align
. . .
[25] .comment PROGBITS 0000000000000000 00002c70
00000000000000b2 0000000000000001 MS 0 0 1
[26] .gnu_debuglink PROGBITS 0000000000000000 00003bc8
0000000000000010 0000000000000000 0 0 1
[27] .shstrtab STRTAB 0000000000000000 00003bd8
0000000000000100 0000000000000000 0 0 1
[28] .symtab SYMTAB 0000000000000000 00002d28
0000000000000ea0 0000000000000018 29 96 8
[29] .strtab STRTAB 0000000000000000 00003cd8
00000000000003b3 0000000000000000 0 0 1

Post by Mark Millard via freebsd-current
00003bd4-00003bc8 == 0xC

Note: 0xC+0x4 == 0x10 (the size), so the last 4 bytes
of .gnu_debuglink

Post by Mark Millard via freebsd-current
# cmp -x /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/bin/chflags /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/bin/chflags
00002208 88 a1
00002209 e6 40
0000220a 60 94
0000220b bf ce
# ls -Tld /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/bin/chflags /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/bin/chflags
-r-xr-xr-x 1 root wheel 11440 May 4 08:55:01 2021 /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/bin/chflags
-r-xr-xr-x 1 root wheel 11440 May 3 23:16:36 2021 /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/bin/chflags
Idx Name Size VMA LMA File off Algn
. . .
25 .gnu_debuglink 00000014 0000000000000000 0000000000000000 000021f8 2**0
CONTENTS, READONLY

Section Headers:
[Nr] Name Type Address Offset
Size EntSize Flags Link Info Align
. . .
[25] .comment PROGBITS 0000000000000000 000016d8
00000000000000b2 0000000000000001 MS 0 0 1
[26] .gnu_debuglink PROGBITS 0000000000000000 000021f8
0000000000000014 0000000000000000 0 0 1
[27] .shstrtab STRTAB 0000000000000000 0000220c
0000000000000100 0000000000000000 0 0 1
[28] .symtab SYMTAB 0000000000000000 00001790
0000000000000a68 0000000000000018 29 83 8
[29] .strtab STRTAB 0000000000000000 0000230c
000000000000021f 0000000000000000 0 0 1

Post by Mark Millard via freebsd-current
00002208-000021f8 == 0x10

Note: 0x10+0x4 == 0x14 (the size), so the last 4 bytes
of .gnu_debuglink

Post by Mark Millard via freebsd-current
# cmp -x /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/bin/chio /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/bin/chio
000050c4 6b 3e
000050c5 08 ca
000050c6 7a 2f
000050c7 5d 64
# ls -Tld /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/bin/chio /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/bin/chio
-r-xr-xr-x 1 root wheel 23728 May 4 08:55:01 2021 /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/bin/chio
-r-xr-xr-x 1 root wheel 23728 May 3 23:16:37 2021 /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/bin/chio
Idx Name Size VMA LMA File off Algn
. . .
25 .gnu_debuglink 00000010 0000000000000000 0000000000000000 000050b8 2**0
CONTENTS, READONLY

Section Headers:
[Nr] Name Type Address Offset
Size EntSize Flags Link Info Align
. . .
[25] .comment PROGBITS 0000000000000000 00004298
00000000000000b2 0000000000000001 MS 0 0 1
[26] .gnu_debuglink PROGBITS 0000000000000000 000050b8
0000000000000010 0000000000000000 0 0 1
[27] .shstrtab STRTAB 0000000000000000 000050c8
0000000000000100 0000000000000000 0 0 1
[28] .symtab SYMTAB 0000000000000000 00004350
0000000000000d68 0000000000000018 29 100 8
[29] .strtab STRTAB 0000000000000000 000051c8
0000000000000363 0000000000000000 0 0 1

Post by Mark Millard via freebsd-current
000050c4-000050b8 == 0xC

Note: 0xC+0x4 == 0x10 (the size), so the last 4 bytes
of .gnu_debuglink

Post by Mark Millard via freebsd-current
For all I know, some individual byte(s) in the 4 might accidentally
match sometimes. The addition offset after .gnu_debuglink's file
offset does vary (0xC and 0x10 above).

Specifically: the last 4 bytes of the .gnu_debuglink section.

Post by Mark Millard via freebsd-current
The content of those differences do not look like
file path components, for example the 0x08 byte.
DEBUG_FLAGS+=
#
WITH_REPRODUCIBLE_BUILD=
WITH_DEBUG_FILES=
But that was true for the earlier armv7 target example
that I reported that only got a few files with
differences.

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

Mark Millard via freebsd-current

2021-05-04 17:31:50 UTC

Permalink

I probably know why the huge count of differences this time
unlike the original report . . .

Previously I built based on a checked-in branch as part of
my experimenting. This time it was in a -dirty form (not
checked in), again as part of my experimental exploration.

WITH_REPRODUCIBLE_BUILD= makes a distinction between these
if I remember right: (partially?) disabling itself for
-dirty style.

To reproduce the original style of test I need to create
a branch with my few patches checked in and do the
buildworlds from that branch.

This will, of course, take a while.

Sorry for the noise.

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

Mark Millard via freebsd-current

2021-05-04 20:38:23 UTC

Permalink

[The first buidlworld is still in process. So while waiting . . .]

Post by Mark Millard via freebsd-current
I probably know why the huge count of differences this time
unlike the original report . . .
Previously I built based on a checked-in branch as part of
my experimenting. This time it was in a -dirty form (not
checked in), again as part of my experimental exploration.
WITH_REPRODUCIBLE_BUILD= makes a distinction between these
if I remember right: (partially?) disabling itself for
-dirty style.
To reproduce the original style of test I need to create
a branch with my few patches checked in and do the
buildworlds from that branch.
This will, of course, take a while.
Sorry for the noise.

I've confirmed some of the details of the large number of
files with difference while waiting for the 1st buildworld :

The 4 bytes at the end of the .gnu_debuglink section
that are ending up different are the checksum for the
.debug file. The .debug files have differences such as:

│ - <1a> DW_AT_comp_dir : (indirect string) /usr/obj/BUILDs/13_0R-CA72-nodbg-clang-alt/usr/13_0R-src/arm64.aarch64/lib/csu/aarch64
│ + <1a> DW_AT_comp_dir : (indirect string) /usr/obj/BUILDs/13_0R-CA72-nodbg-clang/usr/13_0R-src/arm64.aarch64/lib/csu/aarch64

So I need to build, snapshot (in case need
to reference), install, clean-out, build,
install elsewhere, compare. (Or analogous
that uses the same build base-path for both
installs despite separate buildworld's.)
This is separate from any potential -dirty
vs. checked-in handling variation by
WITH_REPRODUCIBLE_BUILD= .

My process that produced the original armv7
report happened to do that before I accidentally
discovered the presence of the few files with
differences. My new experiments were different
and I'd not though of needing to vary the
procedure to get you the right evidence.

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

Mark Millard via freebsd-current

2021-05-05 03:26:30 UTC

Permalink

Post by Mark Millard via freebsd-current
[The first buidlworld is still in process. So while waiting . . .]

I've confirmed some of the details of the large number of
The 4 bytes at the end of the .gnu_debuglink section
that are ending up different are the checksum for the
│ - <1a> DW_AT_comp_dir : (indirect string) /usr/obj/BUILDs/13_0R-CA72-nodbg-clang-alt/usr/13_0R-src/arm64.aarch64/lib/csu/aarch64
│ + <1a> DW_AT_comp_dir : (indirect string) /usr/obj/BUILDs/13_0R-CA72-nodbg-clang/usr/13_0R-src/arm64.aarch64/lib/csu/aarch64
So I need to build, snapshot (in case need
to reference), install, clean-out, build,
install elsewhere, compare. (Or analogous
that uses the same build base-path for both
installs despite separate buildworld's.)
This is separate from any potential -dirty
vs. checked-in handling variation by
WITH_REPRODUCIBLE_BUILD= .
My process that produced the original armv7
report happened to do that before I accidentally
discovered the presence of the few files with
differences. My new experiments were different
and I'd not though of needing to vary the
procedure to get you the right evidence.

The two aarch64 test installs did not show any
differences in a "diff -rq" . Ignoring *.meta
files generated during the builds, the build
directory tree snapshots showed just the
differences:

# diff -rq /usr/obj/BUILDs/13_0R-CA72-nodbg-clang-alt/.zfs/snapshot/commited-style-0/usr /usr/obj/BUILDs/13_0R-CA72-nodbg-clang-alt/.zfs/snapshot/commited-style-1/usr | grep -v '\.meta' | more
Files /usr/obj/BUILDs/13_0R-CA72-nodbg-clang-alt/.zfs/snapshot/commited-style-0/usr/13_0R-src/arm64.aarch64/stand/ficl/softcore.c and /usr/obj/BUILDs/13_0R-CA72-nodbg-clang-alt/.zfs/snapshot/commited-style-1/usr/13_0R-src/arm64.aarch64/stand/ficl/softcore.c differ
Files /usr/obj/BUILDs/13_0R-CA72-nodbg-clang-alt/.zfs/snapshot/commited-style-0/usr/13_0R-src/arm64.aarch64/toolchain-metadata.mk and /usr/obj/BUILDs/13_0R-CA72-nodbg-clang-alt/.zfs/snapshot/commited-style-1/usr/13_0R-src/arm64.aarch64/toolchain-metadata.mk differ

# diff -u /usr/obj/BUILDs/13_0R-CA72-nodbg-clang-alt/.zfs/snapshot/commited-style-0/usr/13_0R-src/arm64.aarch64/stand/ficl/softcore.c /usr/obj/BUILDs/13_0R-CA72-nodbg-clang-alt/.zfs/snapshot/commited-style-1/usr/13_0R-src/arm64.aarch64/stand/ficl/softcore.c
--- /usr/obj/BUILDs/13_0R-CA72-nodbg-clang-alt/.zfs/snapshot/commited-style-0/usr/13_0R-src/arm64.aarch64/stand/ficl/softcore.c 2021-05-04 13:45:14.463351000 -0700
+++ /usr/obj/BUILDs/13_0R-CA72-nodbg-clang-alt/.zfs/snapshot/commited-style-1/usr/13_0R-src/arm64.aarch64/stand/ficl/softcore.c 2021-05-04 19:04:32.338203000 -0700
@@ -4,7 +4,7 @@
** Words from CORE set written in FICL
** Author: John Sadler (***@alum.mit.edu)
** Created: 27 December 1997
-** Last update: Tue May 4 13:45:14 PDT 2021
+** Last update: Tue May 4 19:04:32 PDT 2021
*******************************************************************/
/*
** DO NOT EDIT THIS FILE -- it is generated by softwords/softcore.awk

# diff -u /usr/obj/BUILDs/13_0R-CA72-nodbg-clang-alt/.zfs/snapshot/commited-style-0/usr/13_0R-src/arm64.aarch64/toolchain-metadata.mk /usr/obj/BUILDs/13_0R-CA72-nodbg-clang-alt/.zfs/snapshot/commited-style-1/usr/13_0R-src/arm64.aarch64/toolchain-metadata.mk
--- /usr/obj/BUILDs/13_0R-CA72-nodbg-clang-alt/.zfs/snapshot/commited-style-0/usr/13_0R-src/arm64.aarch64/toolchain-metadata.mk 2021-05-04 10:55:26.030179000 -0700
+++ /usr/obj/BUILDs/13_0R-CA72-nodbg-clang-alt/.zfs/snapshot/commited-style-1/usr/13_0R-src/arm64.aarch64/toolchain-metadata.mk 2021-05-04 16:14:24.513346000 -0700
@@ -1,4 +1,4 @@
-.info Using cached toolchain metadata from build at CA72_4c8G_ZFS on Tue May 4 10:55:26 PDT 2021
+.info Using cached toolchain metadata from build at CA72_4c8G_ZFS on Tue May 4 16:14:24 PDT 2021
_LOADED_TOOLCHAIN_METADATA=t
COMPILER_VERSION=110001
X_COMPILER_VERSION=110001

I may run a 'target cortex-a7 (so: armv7) from aarch64' test
next. That was the context for the original discovery and
report.

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

Mark Millard via freebsd-current

2021-05-05 13:09:59 UTC

Permalink

Post by Mark Millard via freebsd-current

Post by Mark Millard via freebsd-current
[The first buidlworld is still in process. So while waiting . . .]

The two aarch64 test installs did not show any
differences in a "diff -rq" . Ignoring *.meta
files generated during the builds, the build
directory tree snapshots showed just the
# diff -rq /usr/obj/BUILDs/13_0R-CA72-nodbg-clang-alt/.zfs/snapshot/commited-style-0/usr /usr/obj/BUILDs/13_0R-CA72-nodbg-clang-alt/.zfs/snapshot/commited-style-1/usr | grep -v '\.meta' | more
Files /usr/obj/BUILDs/13_0R-CA72-nodbg-clang-alt/.zfs/snapshot/commited-style-0/usr/13_0R-src/arm64.aarch64/stand/ficl/softcore.c and /usr/obj/BUILDs/13_0R-CA72-nodbg-clang-alt/.zfs/snapshot/commited-style-1/usr/13_0R-src/arm64.aarch64/stand/ficl/softcore.c differ
Files /usr/obj/BUILDs/13_0R-CA72-nodbg-clang-alt/.zfs/snapshot/commited-style-0/usr/13_0R-src/arm64.aarch64/toolchain-metadata.mk and /usr/obj/BUILDs/13_0R-CA72-nodbg-clang-alt/.zfs/snapshot/commited-style-1/usr/13_0R-src/arm64.aarch64/toolchain-metadata.mk differ
# diff -u /usr/obj/BUILDs/13_0R-CA72-nodbg-clang-alt/.zfs/snapshot/commited-style-0/usr/13_0R-src/arm64.aarch64/stand/ficl/softcore.c /usr/obj/BUILDs/13_0R-CA72-nodbg-clang-alt/.zfs/snapshot/commited-style-1/usr/13_0R-src/arm64.aarch64/stand/ficl/softcore.c
--- /usr/obj/BUILDs/13_0R-CA72-nodbg-clang-alt/.zfs/snapshot/commited-style-0/usr/13_0R-src/arm64.aarch64/stand/ficl/softcore.c 2021-05-04 13:45:14.463351000 -0700
+++ /usr/obj/BUILDs/13_0R-CA72-nodbg-clang-alt/.zfs/snapshot/commited-style-1/usr/13_0R-src/arm64.aarch64/stand/ficl/softcore.c 2021-05-04 19:04:32.338203000 -0700
@@ -4,7 +4,7 @@
** Words from CORE set written in FICL
** Created: 27 December 1997
-** Last update: Tue May 4 13:45:14 PDT 2021
+** Last update: Tue May 4 19:04:32 PDT 2021
*******************************************************************/
/*
** DO NOT EDIT THIS FILE -- it is generated by softwords/softcore.awk
# diff -u /usr/obj/BUILDs/13_0R-CA72-nodbg-clang-alt/.zfs/snapshot/commited-style-0/usr/13_0R-src/arm64.aarch64/toolchain-metadata.mk /usr/obj/BUILDs/13_0R-CA72-nodbg-clang-alt/.zfs/snapshot/commited-style-1/usr/13_0R-src/arm64.aarch64/toolchain-metadata.mk
--- /usr/obj/BUILDs/13_0R-CA72-nodbg-clang-alt/.zfs/snapshot/commited-style-0/usr/13_0R-src/arm64.aarch64/toolchain-metadata.mk 2021-05-04 10:55:26.030179000 -0700
+++ /usr/obj/BUILDs/13_0R-CA72-nodbg-clang-alt/.zfs/snapshot/commited-style-1/usr/13_0R-src/arm64.aarch64/toolchain-metadata.mk 2021-05-04 16:14:24.513346000 -0700
@@ -1,4 +1,4 @@
-.info Using cached toolchain metadata from build at CA72_4c8G_ZFS on Tue May 4 10:55:26 PDT 2021
+.info Using cached toolchain metadata from build at CA72_4c8G_ZFS on Tue May 4 16:14:24 PDT 2021
_LOADED_TOOLCHAIN_METADATA=t
COMPILER_VERSION=110001
X_COMPILER_VERSION=110001
I may run a 'target cortex-a7 (so: armv7) from aarch64' test
next. That was the context for the original discovery and
report.

The armv7 builds and installs get the same sort of diff
results as the aarch64 ones.

So it does not look like I can readily reproduce the
problem files that had differences.

Given the time it takes to run the experiments, I would
not expect to reproduce the problem on any given
timescale. I'll stop the reporting as I go along in
my activities --unless I end up with a repetition
of the problem.

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)