setting CFLAGS in /etc/make.conf

[Oliver Eikemeier]

I have a problem with the following recommendation in
src/share/examples/etc/make.conf:

# CFLAGS controls the compiler settings used when compiling C code.
# Note that optimization settings other than -O and -O2 are not
recommended
# or supported for compiling the world or the kernel - please revert any
# nonstandard optimization settings to "-O" before submitting bug reports
# without patches to the developers.
# Note also that at this time the -O2 setting is known to expose bugs in
# libalias(3), and possibly other parts of the system.
#
#CFLAGS= -O -pipe

Basically, when a port tries to pass CFLAGS to the distribution
Makefile, this is done by
env CFLAGS="${CFLAGS}" make
which is effectively overwritten by users setting CFLAGS in
make.conf(5). OTOH when += is used, it inherits the previous values, so
when someone would do
CFLAGS+= -O -pipe
in make.conf(5) the effective value passed is "-O -pipe -O -pipe
${_CPUCFLAGS}", which is not harmful, but not good either. This is
especially problematic when the port uses CFLAGS="${PTHREAD_CFLAGS}" or
CFLAGS="-DMY_OPTION", and a lot of ports use CFLAGS. This is no problem
when the port has USE_GMAKE, since gmake doesn't read make.conf(5).


The following Makefile illustrates the problem:

all:
@env CFLAGS="${CFLAGS} -DFOO" ${MAKE} cflags

.ifmake cflags
CFLAGS+= -DBAR
.endif

cflags:
@${ECHO} "CFLAGS=${CFLAGS}"

with the output:

CFLAGS=-O -pipe ${_CPUCFLAGS} -DFOO ${_CPUCFLAGS} -DBAR

Basically this is the expected result, although the ${_CPUCFLAGS} is
unfortunately doubled. This is what a typical port would see. When I now
set

CFLAGS= -O -pipe

in make.conf(5), which is the example given in
share/examples/etc/make.conf, I get:

CFLAGS=-O -pipe ${_CPUCFLAGS} -DBAR

so I loose the settings done in the ports Makefile (-DFOO).

One solution would be to set __MAKE_CONF=/dev/null in bsd.port.mk,
another solution is to deprecate setting CFLAGS in make.conf(5), or use
an alternate variable.

Generally I believe it is unexpected that an reexecution of make(1)
overwrites CFLAGS explicitly passed in the environment, so deprecating
its usage is my favorite option here. Btw, passing CFLAGS in the command
line is not an option, since Makefiles need to be able to add to the
value.

Thoughs?
-Oliver

[Ruslan Ermilov]

Post by Oliver Eikemeier
I have a problem with the following recommendation in
# CFLAGS controls the compiler settings used when compiling C code.
# Note that optimization settings other than -O and -O2 are not
recommended
# or supported for compiling the world or the kernel - please revert any
# nonstandard optimization settings to "-O" before submitting bug reports
# without patches to the developers.
# Note also that at this time the -O2 setting is known to expose bugs in
# libalias(3), and possibly other parts of the system.
#
#CFLAGS= -O -pipe
Basically, when a port tries to pass CFLAGS to the distribution
Makefile, this is done by
env CFLAGS="${CFLAGS}" make
which is effectively overwritten by users setting CFLAGS in
make.conf(5). OTOH when += is used, it inherits the previous values, so
when someone would do
CFLAGS+= -O -pipe
in make.conf(5) the effective value passed is "-O -pipe -O -pipe
${_CPUCFLAGS}", which is not harmful, but not good either. This is
especially problematic when the port uses CFLAGS="${PTHREAD_CFLAGS}" or
CFLAGS="-DMY_OPTION", and a lot of ports use CFLAGS. This is no problem
when the port has USE_GMAKE, since gmake doesn't read make.conf(5).
@env CFLAGS="${CFLAGS} -DFOO" ${MAKE} cflags
.ifmake cflags
CFLAGS+= -DBAR
.endif
@${ECHO} "CFLAGS=${CFLAGS}"
CFLAGS=-O -pipe ${_CPUCFLAGS} -DFOO ${_CPUCFLAGS} -DBAR
Basically this is the expected result, although the ${_CPUCFLAGS} is
unfortunately doubled.

No, this is an abuse of the intended CFLAGS usage. CFLAGS is
the add-only variable by design. You shouldn't reuse its
value to reinitialize it.

Post by Oliver Eikemeier
This is what a typical port would see. When I now
set
CFLAGS= -O -pipe
in make.conf(5), which is the example given in
CFLAGS=-O -pipe ${_CPUCFLAGS} -DBAR
so I loose the settings done in the ports Makefile (-DFOO).

Just to make the picture clear for the rest of the readers.
This has nothing to do with recent changes to make(1), this
behavior of CFLAGS was forever. Since global variables are
of a higher precedence than envieronment variables, the
unconditional CFLAGS setting from /etc/make.conf takes
precedence of CFLAGS specified in environment. That
explains what you're seeing (and you well understand it,
I'm sure).

Post by Oliver Eikemeier
One solution would be to set __MAKE_CONF=/dev/null in bsd.port.mk,
another solution is to deprecate setting CFLAGS in make.conf(5), or use
an alternate variable.

Both aren't good. People may have something set in /etc/make.conf
for a good reason, e.g. to influence the build of some ports, for
example. Also, you cannot "deprecate" unconditional (with the `='
operator) setting of CFLAGS in /etc/make.conf, as this is a well
established practice, so there will always be someone who sets it
this way.

Post by Oliver Eikemeier
Generally I believe it is unexpected that an reexecution of make(1)
overwrites CFLAGS explicitly passed in the environment, so deprecating
its usage is my favorite option here. Btw, passing CFLAGS in the command
line is not an option, since Makefiles need to be able to add to the
value.
Thoughs?

I suggest another solution: modifying the distribution makefile
when needed, by adding the CFLAGS+=-DFOO onto the top of it, or
doing something equivalent. This works most reliable:

Port's main makefile:

: all:
: # commands that modify the distribution makefile
: @${MAKE} cflags

Distribution makefile:

: CFLAGS+= -DFOO # this line was added by the port's makefile
: .ifmake cflags
: CFLAGS+= -DBAR
: .endif
:
: cflags:
: @${ECHO} "CFLAGS=${CFLAGS}"

An equivalent thing that doesn't involve modifying the distribution
makefile would be:

Port's main makefile:

: all:
: @cd ${WRKDIR} && ${MAKE} -f Makefile.wrapper cflags

Ports produced makefile (Makefile.wrapper) in WRKDIR:

: CFLAGS+= -DFOO
: .include "Makefile"

Distribution's original makefile (Makefile):

: .ifmake cflags
: CFLAGS+= -DBAR
: .endif
:
: cflags:
: @${ECHO} "CFLAGS=${CFLAGS}"


Cheers,

[Oliver Eikemeier]

Ruslan Ermilov wrote:

[...]

Post by Ruslan Ermilov

Post by Oliver Eikemeier
@env CFLAGS="${CFLAGS} -DFOO" ${MAKE} cflags
.ifmake cflags
CFLAGS+= -DBAR
.endif
@${ECHO} "CFLAGS=${CFLAGS}"
CFLAGS=-O -pipe ${_CPUCFLAGS} -DFOO ${_CPUCFLAGS} -DBAR
Basically this is the expected result, although the ${_CPUCFLAGS} is
unfortunately doubled.

No, this is an abuse of the intended CFLAGS usage. CFLAGS is
the add-only variable by design. You shouldn't reuse its
value to reinitialize it.

This is the current practice in bsd.port.mk for ages. It may be wrong,
although it works when the port uses gmake(1).

Post by Ruslan Ermilov

Post by Oliver Eikemeier
This is what a typical port would see. When I now
set
CFLAGS= -O -pipe
in make.conf(5), which is the example given in
CFLAGS=-O -pipe ${_CPUCFLAGS} -DBAR
so I loose the settings done in the ports Makefile (-DFOO).

Just to make the picture clear for the rest of the readers.
This has nothing to do with recent changes to make(1), this
behavior of CFLAGS was forever.

Yes, sorry that I didn't made this explicit. I was not sure whether this
belongs to current@ or arch@, so I just choosed one. Probably we need to
change _something_, so it's not the wrong place.

Post by Ruslan Ermilov
Since global variables are
of a higher precedence than envieronment variables, the
unconditional CFLAGS setting from /etc/make.conf takes
precedence of CFLAGS specified in environment. That
explains what you're seeing (and you well understand it,
I'm sure).

Yup. Although it is unexpected, especially contrasted to the behaviour
when using gmake(1). From a ports point of view, of course.

Post by Ruslan Ermilov

Post by Oliver Eikemeier
One solution would be to set __MAKE_CONF=/dev/null in bsd.port.mk,
another solution is to deprecate setting CFLAGS in make.conf(5), or use
an alternate variable.

Both aren't good. People may have something set in /etc/make.conf
for a good reason, e.g. to influence the build of some ports, for
example.

You will only influence ports that use make(1), which is not obvious on
the first glance. Even then, an update might switch to gmake(1) because
the Makefile suddenly uses gmake(1) features, or it may switch to
make(1), because the ports maintainer might have recognized that the
dependency on gmake(1) is unnecessary, so you can never be sure.

Not that the ports Makefile is always run through make(1), so
bsd.port.mk passes CFLAGS from /etc/make.conf through to the
distributions Makefile, no matter whether it is build with make(1),
gmake(1) or something else.

Post by Ruslan Ermilov
Also, you cannot "deprecate" unconditional (with the `='
operator) setting of CFLAGS in /etc/make.conf, as this is a well
established practice, so there will always be someone who sets it
this way.

Ok, but there must be some way to fix it.

Post by Ruslan Ermilov

Post by Oliver Eikemeier
Generally I believe it is unexpected that an reexecution of make(1)
overwrites CFLAGS explicitly passed in the environment, so deprecating
its usage is my favorite option here. Btw, passing CFLAGS in the command
line is not an option, since Makefiles need to be able to add to the
value.
Thoughs?

I suggest another solution: modifying the distribution makefile
when needed, by adding the CFLAGS+=-DFOO onto the top of it, or

[...]

Some problems are:

- The CFLAGS needed might be dynamic (depending on configuration
values), so it is not easy to patch (although you could use
CFLAGS+=${PORTCFLAGS} and set that variable instead)

- The CFLAGS might be needed in the configuration phase

- The port might use gmake(1), so it won't pick up the values from
/etc/make.conf by itself (when not passed explicitly)

- The port might have multiple (sub-)makefiles, it is hard to patch them
all

- The port might pass CFLAGS to submakes by itself

- The port might not have any Makefiles, but use a different method to
build

Post by Ruslan Ermilov
An equivalent thing that doesn't involve modifying the distribution

This has basically the same problems as above.

The point is: when you currently set CFLAGS in make.conf(5), some ports
will break. It's not easy to do an survey which ports are affected since
many use the value given to configure, hardcoded as CFLAGS=... in the
generated Makefiles, in fact most do it. One sample where this fails is
net/obnc, which misses -DIPSEC.

I have the feeling I open a can of worms here. The bug seems to appear
rarely (since most ports don't care which CFLAGS are passed during
building, they just use the ones from configuring), but it is nasty to
track down. Plus, when you think the base system is correct, we must
change something in ports...

-Oliver

[Garance A Drosihn]

Post by Oliver Eikemeier
Basically, when a port tries to pass CFLAGS to the distribution
Makefile, this is done by
env CFLAGS="${CFLAGS}" make
which is effectively overwritten by users setting CFLAGS in
make.conf(5). OTOH when += is used, it inherits the previous
values, so when someone would do
CFLAGS+= -O -pipe
in make.conf(5) the effective value passed is
"-O -pipe -O -pipe ${_CPUCFLAGS}",
which is not harmful, but not good either. This is especially
problematic when the port uses CFLAGS="${PTHREAD_CFLAGS}" or
CFLAGS="-DMY_OPTION", and a lot of ports use CFLAGS. This is
no problem when the port has USE_GMAKE, since gmake doesn't
read make.conf(5).

Back in March I was working on an updated version of the net/cap
port, and I noticed this problem with CFLAGS in /etc/make.conf vs
CFLAGS in the makefile of a port. (So I am pretty sure this issue
has nothing to do with the recent changes to `make'.)

What I ended up doing was changing the net/cap/Makefile to set
CAP_CFLAGS instead of CFLAGS. The distributed net/cap source has
a bunch of Makefile.m4 files which are used to generate the real
makefiles, and those Makefile.m4 files were already adding a bunch
of values to CFLAGS. So, I changed a line (in net/cap/Makefile)
from:

${FIND} ${WRKSRC} -name Makefile.m4 \
-exec ${REINPLACE_CMD} \
-e 's/CFLAGS=/CFLAGS+=/' \{\} \;

to:

${FIND} ${WRKSRC} -name Makefile.m4 \
-exec ${REINPLACE_CMD} \
-e "s/CFLAGS=/CFLAGS+=${CAP_CFLAGS} /" \{\} \;

I meant to follow up on this issue on the freebsd-ports mailing
list, but I guess I never got around to mentioning it. As near as
I could tell, it is just a bad idea for a ports-makefile to depend
on setting CFLAGS. The fix I went with for net/cap probably will
not make sense for many other ports, but it was the easy fix based
on what net/cap/Makefile was already doing.

[note that I have not sent in that updated CAP port. What I have
seems to work fine on FreeBSD-5.x on i386, but I wanted to see if
I could get it working on sparc64, and to also include some bug
fixes that I have made to cap at RPI]

[Oliver Eikemeier]

Post by Garance A Drosihn
Back in March I was working on an updated version of the net/cap
port, and I noticed this problem with CFLAGS in /etc/make.conf vs
CFLAGS in the makefile of a port. (So I am pretty sure this issue
has nothing to do with the recent changes to `make'.)

Sure. I even tested it with make(1) from 4.7. This has been there for
ages.

Post by Garance A Drosihn
What I ended up doing was changing the net/cap/Makefile to set
CAP_CFLAGS instead of CFLAGS. The distributed net/cap source has
a bunch of Makefile.m4 files which are used to generate the real
makefiles, and those Makefile.m4 files were already adding a bunch
of values to CFLAGS. So, I changed a line (in net/cap/Makefile)
${FIND} ${WRKSRC} -name Makefile.m4 \
-exec ${REINPLACE_CMD} \
-e 's/CFLAGS=/CFLAGS+=/' \{\} \;
${FIND} ${WRKSRC} -name Makefile.m4 \
-exec ${REINPLACE_CMD} \
-e "s/CFLAGS=/CFLAGS+=${CAP_CFLAGS} /" \{\} \;

Jup, but this is hard to generalize.

Post by Garance A Drosihn
I meant to follow up on this issue on the freebsd-ports mailing
list, but I guess I never got around to mentioning it. As near as
I could tell, it is just a bad idea for a ports-makefile to depend
on setting CFLAGS.

Many do, but most just hardcode the CFLAGS value they see in the
configuration step in the (generated) ports Makefile.

I run into this recently with mail/exim. Of course it was possible to
work around this too, but it happened only when certain configuration
options where given *and* CFLAGS was set in /etc/make.conf. There must
be a better way to deal with this.

-Oliver

[Ruslan Ermilov]

Post by Oliver Eikemeier
I run into this recently with mail/exim. Of course it was possible to
work around this too, but it happened only when certain configuration
options where given *and* CFLAGS was set in /etc/make.conf. There must
be a better way to deal with this.

Facts:

1. In FreeBSD, CFLAGS (if not explicitly set, or set in /etc/make.conf)
is the global make(1) variable.

2. FreeBSD make(1) knows about environment and global variables, and
global variables take precedence over environment variables.

3. If CFLAGS is not explicitly set in /etc/make.conf, and CFLAGS is
set in environment instead, its value becomes a value of the CFLAGS
make(1) global variable:

: $ cat makefile
: FOO+= bar
:
: all:
: @echo global FOO: ${FOO}
: @echo env FOO: $${FOO}
: $ FOO=foo make
: global FOO: foo bar
: env FOO: foo

So, if you need to change another makefile's idea of the initial value
of CFLAGS, you basically have two reliable choices:

a) Don't use /etc/make.conf to avoid the possibility of setting
CFLAGS in /etc/make.conf.

b) Modify this another makefile to add things you want to CFLAGS;
the modification may be either hardcoded, or using another
macro whose value you can then pass as environment variable.

There's no other reliable way, and FreeBSD make(1) doesn't provide
you a way to initialize a variable in the global context on the
command line or from environment, except for -D which would set it
to "1". You can only modify global variables from makefile or
from its included sources.

P.S. I start to hate command-line variable in make(1). ;)


Cheers,

[Harti Brandt]

On Fri, 20 Aug 2004, Ruslan Ermilov wrote:

RE>On Fri, Aug 20, 2004 at 02:08:33PM +0200, Oliver Eikemeier wrote:
RE>> I run into this recently with mail/exim. Of course it was possible to
RE>> work around this too, but it happened only when certain configuration
RE>> options where given *and* CFLAGS was set in /etc/make.conf. There must
RE>> be a better way to deal with this.
RE>>
RE>Facts:
RE>
RE>1. In FreeBSD, CFLAGS (if not explicitly set, or set in /etc/make.conf)
RE> is the global make(1) variable.
RE>
RE>2. FreeBSD make(1) knows about environment and global variables, and
RE> global variables take precedence over environment variables.
RE>
RE>3. If CFLAGS is not explicitly set in /etc/make.conf, and CFLAGS is
RE> set in environment instead, its value becomes a value of the CFLAGS
RE> make(1) global variable:
RE>
RE>: $ cat makefile
RE>: FOO+= bar
RE>:
RE>: all:
RE>: @echo global FOO: ${FOO}
RE>: @echo env FOO: $${FOO}
RE>: $ FOO=foo make
RE>: global FOO: foo bar
RE>: env FOO: foo
RE>
RE>So, if you need to change another makefile's idea of the initial value
RE>of CFLAGS, you basically have two reliable choices:
RE>
RE>a) Don't use /etc/make.conf to avoid the possibility of setting
RE> CFLAGS in /etc/make.conf.
RE>
RE>b) Modify this another makefile to add things you want to CFLAGS;
RE> the modification may be either hardcoded, or using another
RE> macro whose value you can then pass as environment variable.
RE>
RE>There's no other reliable way, and FreeBSD make(1) doesn't provide
RE>you a way to initialize a variable in the global context on the
RE>command line or from environment, except for -D which would set it
RE>to "1". You can only modify global variables from makefile or
RE>from its included sources.
RE>
RE>P.S. I start to hate command-line variable in make(1). ;)

They have their place so you could do make CC=icc without caring what
Makefile does try to set CC. You just need to use them for the things they
are intended for.

The problem with CFLAGS is that it is overloaded to be a variable that
should be twiddled by the user, by make (via .mk and make.conf) and by
individual makefiles. This is normally (read on other systems) not the
case - CFLAGS used to be a variable used by makefiles. I think setting
CFLAGS from make.conf is bad.

<THINKING LOUD>

make.conf should probably have a different variable (something like
CFLAGS_DEFAULT or whatever) that just gets applied at the places where
CFLAGS is used to pass them to a command. This would somewhat (but not
entirely) decouple the different CFLAGS uses. Another step would be to
have CFLAGS_SYS (or whatever) that gets set in *.mk. So we would end up
with

${CC} ${CFLAGS} ${CFLAGS_DEFAULT} ${CFLAGS_SYS} ...

and everybody could do with CFLAGS whatever he needs to do.

</THINKING LOUD>

harti

[horio shoichi]

On Fri, 20 Aug 2004 15:36:05 +0300

Post by Ruslan Ermilov
1. In FreeBSD, CFLAGS (if not explicitly set, or set in /etc/make.conf)
is the global make(1) variable.
2. FreeBSD make(1) knows about environment and global variables, and
global variables take precedence over environment variables.
3. If CFLAGS is not explicitly set in /etc/make.conf, and CFLAGS is
set in environment instead, its value becomes a value of the CFLAGS
: $ cat makefile
: FOO+= bar
: $ FOO=foo make
: global FOO: foo bar
: env FOO: foo
So, if you need to change another makefile's idea of the initial value
a) Don't use /etc/make.conf to avoid the possibility of setting
CFLAGS in /etc/make.conf.
b) Modify this another makefile to add things you want to CFLAGS;
the modification may be either hardcoded, or using another
macro whose value you can then pass as environment variable.
There's no other reliable way, and FreeBSD make(1) doesn't provide
you a way to initialize a variable in the global context on the
command line or from environment, except for -D which would set it
to "1". You can only modify global variables from makefile or
from its included sources.
P.S. I start to hate command-line variable in make(1). ;)
Cheers,
--
Ruslan Ermilov
FreeBSD committer

I think following way is "reliable". No ?

% grep CFLAGS /etc/make.conf
CFLAGS ?= -O -pipe
%

or equivalently, (more polite and maybe more robust)

% grep CFLAGS /etc/make.conf
.if defined(ENV_CFLAGS) && !empty(ENV_CFLAGS)
CFLAGS = ${ENV_CFLAGS}
.else # CFLAGS
CFLAGS = -O -pipe
.endif # CFLAGS
%

I myself is using a variation of the latter for ports,
to occasionally set -fno-stack-protector.


horio shoichi

[horio shoichi]

On Fri, 20 Aug 2004 15:36:05 +0300

Post by Ruslan Ermilov
P.S. I start to hate command-line variable in make(1). ;)
Cheers,
--
Ruslan Ermilov
FreeBSD committer

I forgot about command line variations.

% cat cflags
.PHONY : a
a :
# ${CFLAGS}
make -f cflags dee-dee
dee-dee :
# ${CFLAGS}
% make -f cflags
# -O -pipe
make -f cflags dee-dee
# -O -pipe
% env CFLAGS="do-do" make -f cflags -E CFLAGS
# do-do
make -f cflags dee-dee
# do-do
%

And, amazing ! featuristic !

% make -f cflags -E CFLAGS CFLAGS=hehe
# hehe
make -f cflags dee-dee
# hehe
% echo $CFLAGS
CFLAGS: Undefined variable.


If the makefiles in nest don't have conflicting variable
requirements, then passing variable values over to the lower
strata makefiles seems not hard.


BTW,

o tested on 4.9-STABLE,
o /etc/make.conf is not mocked up one (as I wrote in
previous text).


horio shoichi

[Garance A Drosihn]

Post by Oliver Eikemeier

Post by Garance A Drosihn
What I ended up doing was changing the net/cap/Makefile to set
CAP_CFLAGS instead of CFLAGS. The distributed net/cap source has
a bunch of Makefile.m4 files which are used to generate the real
makefiles, and those Makefile.m4 files were already adding a bunch
of values to CFLAGS. So, I changed a line (in net/cap/Makefile)
${FIND} ${WRKSRC} -name Makefile.m4 \
-exec ${REINPLACE_CMD} \
-e 's/CFLAGS=/CFLAGS+=/' \{\} \;
${FIND} ${WRKSRC} -name Makefile.m4 \
-exec ${REINPLACE_CMD} \
-e "s/CFLAGS=/CFLAGS+=${CAP_CFLAGS} /" \{\} \;

Jup, but this is hard to generalize.

I think it would be easy to generalize, although for most ports
you probably have to create a patch for the Makefile(s) instead
of using a convenient ${REINPLACE_CMD} command.

In general:
the port makefile should set <portname>_CFLAGS (like "CAP_CFLAGS")
and then change the makefiles distributed with the package
so they have the line "CFLAGS+=<portname>_CFLAGS"

Post by Oliver Eikemeier

Post by Garance A Drosihn
I meant to follow up on this issue on the freebsd-ports mailing
list, but I guess I never got around to mentioning it. As near as
I could tell, it is just a bad idea for a ports-makefile to depend
on setting CFLAGS.

Many do, but most just hardcode the CFLAGS value they see in the
configuration step in the (generated) ports Makefile.

This has the perhaps-unwanted side-effect that you have to 'make clean'
a port any time you change CFLAGS in /etc/make.conf, and you want the
port to pick up that change. That will probably be fine for most users
in most occasions, but occasionally it may cause some confusion.