Resources rebuilt unncessarily during install or on subsequent builds.

[sergey-239]

Dear Joseph,
for some reason, the install target triggers rebuild of build/resources.os, which in turn makes the default target out of date.
Could you please change this behavior, so when there is no actual need to rebuild the default target (it was just successfully built), it is not linked again by 'scons install' invocation.

[jcoffland]

Does my recent commit fix this?

[sergey-239]

Joseph,

unfortunately, no. Building of build/resources.os is still performed in install target, so the default targets get rebuilt.
Thank you,
Sergey

[sergey-239]

Joseph.
after some digging into the problem I found that your patch is actually prevents the recreation of generated files. Though I would use more explicit test for install target like 'install' in BUILD_TARGETS. Nevertheless scons decides for some reason that the build/resources.os deps are changed and compiles it.
The solution is to invoke scons install with --implicit-deps-unchanged option switch: then it does not compile/build anything even without this commit.
For the record I am using python3-scons-4.4.0-2.fc38.noarch

[jcoffland]

I'm not sure why --implicit-deps-unchanged is necessary for your build. It must have something to do with your particular configuration. When I run scons install. No files are rebuilt. The default build is a static build, maybe that has something to do with it.

[sergey-239]

Joseph,
I don't know either. I have a) very basic knowledge of python and b) I need to study the scons source to get to the roots of the problem. Unfortunately I have no time for this now. Presently I am building shared library, but I was encountering the same effect when I was building camotics with static cbang during install build of the camotics. Now I have a workaround, Probably, I would look into this issue some day.
Regards,
Sergey

[sergey-239]

Joseph, I have the hypotesy.
During default target build, scons prints out the following from resource tool emitter:
('build/resources.cpp', 'src/resources.data/data2.cpp')
During install target build, scons outputs this:
('build/resources.cpp', 'build/resources.data/data2.cpp')
This essentially means that the the nodes graph is different between two scons invocations, and scons has no cached time/checksum for node build/resources.data/data2.cpp as a dependency of 'build/resources.cpp', so it marks the latter as need to rebuild node.

Then, the next thing (just after the default target build):

ls -l --time-style=+%s build/resources.cpp src/resources.data/data2.cpp build/resources.data/data2.cpp
ls: cannot access 'src/resources.data/data2.cpp': No such file or directory
-rw-r--r--. 1 mockbuild mock  686 1663962660 build/resources.cpp
-rw-r--r--. 1 mockbuild mock 4469 1663962660 build/resources.data/data2.cpp

this explains everything, but the fact that you do not observe the similar behavior.

Unfortunately, I am not capable to fix this properly in a reasonable time: despite of my thirty five years experience in sw engineering, the functional languages are above my procedural way of thinking...

Regards,
Sergey

Edit:
for your reference, I use two commands:
scons staticlib=0 sharedlib=1 soname=libcbang0.so.1.7.0 with_openssl=0 v8_compress_pointers=0 cxxstd=c++17 debug=1 strict=0 libsuffix=64 'disable_local=zlib bzip2 lz4 sqlite3 expat boost libevent re2 libyaml'
and
scons install prefix=%{buildroot}%{_prefix} staticlib=0 sharedlib=1 soname=libcbang0.so.1.7.0 with_openssl=0 v8_compress_pointers=0 cxxstd=c++17 debug=1 strict=0 libsuffix=64 'disable_local=zlib bzip2 lz4 sqlite3 expat boost libevent re2 libyaml'
(I stripped out the compiler and linker flags for clarity)

[jcoffland]

Yeah src/resources.data/data2.cpp does not and should not exist. I'm not sure how that got in there. When I build with your command line it prints this:

('build/resources.cpp', 'build/resources.data/data2.cpp', 'build/resources.data/data3.cpp', 'build/resources.data/data4.cpp', 'build/resources.data/data5.cpp', 'build/resources.data/data6.cpp', 'build/resources.data/data7.cpp', 'build/resources.data/data8.cpp', 'build/resources.data/data9.cpp', 'build/resources.data/data10.cpp', 'build/resources.data/data11.cpp')

Did you remove the license files from src/resources/licenses/? Maybe I should add an option to just disable the compiled in resources. The only reason they are there in cbang is to be able to display these license files.

[sergey-239]

Ah, yes, I forgot to mention that any third-party code is removed in the %prep phase, so there is only two subdirectories under src/: cbang and resources. also, all licenses from /resources/licensed removed except cbang.txt. This is to ensure that no third-party code is used in build
Yeah src/resources.data/data2.cpp does not and should not exist
It does not exist, see the output of ls command. The emitter returns this result for some reason.

[sergey-239]

Maybe I should add an option to just disable the compiled in resources. The only reason they are there in cbang is to be able to display these license files.

Yes, I think that if lib source is disabled by disable_local, then appropriate license should not be included as well. The reason to include license texts at all is obvious.

[sergey-239]

Two more things to start the install target build using commands I posted above:

1. You need to create a ./lib/libcbang0.so symlink to /.lib/cbang0.so.1.7.0
2. You need to export CBANG_LIBNAME=cbang0
   to let scons find the built library

[sergey-239]

Joseph,

#99 does not fix the issue with wrong deps list yet:
default target:
('build/resources.cpp', 'src/resources.data/data2.cpp', 'src/resources.data/data3.cpp')
install target:
('build/resources.cpp', 'build/resources.data/data2.cpp', 'build/resources.data/data3.cpp')

Regards,
Sergey

[sergey-239]

Joseph,

The problem's source is identified.
Unless the build/resources.data/data2.cpp exists, the File('%s/data%d.cpp' % (data_dir, id)) at config/resources/init: returns the object with path modified to src/....
You do not observe the same behavior as you most likely start the default build from dirty source tree while mock does this always from scratch.
To simulate this behavior just remove build/resources.data and scons cache file before invoking scons.
Also, there is some strange thing with build/resources.data/data2.cpp's dependencies:
a) with or without my patch #99 and some(all) disabled libs, the deps are all files existing in the src/resources/ subtree;
b) the deps are listed three times contrary to regular source files in src/ subtree:

  +-build/resources.data/data2.os
  | +-build/resources.data/data2.cpp
  | | +-src/resources
  | | | +-src/resources/licenses
  | | | | +-src/resources/licenses/boost.txt
  | | | | +-src/resources/licenses/bzip2.txt
  | | | | +-src/resources/licenses/cbang.txt
  | | | | +-src/resources/licenses/expat.txt
  | | | | +-src/resources/licenses/libevent.txt
  | | | | +-src/resources/licenses/libyaml.txt
  | | | | +-src/resources/licenses/lz4.txt
  | | | | +-src/resources/licenses/re2.txt
  | | | | +-src/resources/licenses/sqlite.txt
  | | | | +-src/resources/licenses/test.backup.file~
  | | | | +-src/resources/licenses/zlib.txt
  | | | +-src/resources/test.backup.file~
  | | +-src/cbang/util/Resource.h
  | | +-src/resources/licenses
  | | | +-src/resources/licenses/boost.txt
  | | | +-src/resources/licenses/bzip2.txt
  | | | +-src/resources/licenses/cbang.txt
  | | | +-src/resources/licenses/expat.txt
  | | | +-src/resources/licenses/libevent.txt
  | | | +-src/resources/licenses/libyaml.txt
  | | | +-src/resources/licenses/lz4.txt
  | | | +-src/resources/licenses/re2.txt
  | | | +-src/resources/licenses/sqlite.txt
  | | | +-src/resources/licenses/test.backup.file~
  | | | +-src/resources/licenses/zlib.txt
  | | +-src/resources/test.backup.file~
  | | +-src/resources/licenses/boost.txt
  | | +-src/resources/licenses/bzip2.txt
  | | +-src/resources/licenses/cbang.txt
  | | +-src/resources/licenses/expat.txt
  | | +-src/resources/licenses/libevent.txt
  | | +-src/resources/licenses/libyaml.txt
  | | +-src/resources/licenses/lz4.txt
  | | +-src/resources/licenses/re2.txt
  | | +-src/resources/licenses/sqlite.txt
  | | +-src/resources/licenses/test.backup.file~
  | | +-src/resources/licenses/zlib.txt
  | +-src/cbang/util/Resource.h
  

One more confirmation (from scons --debug=explain install):

('build/resources.cpp', 'build/resources.data/data2.cpp')
scons: rebuilding `build/resources.os' because:
           `src/cbang/util/Resource.h' is no longer a dependency
           `src/cbang/Exception.h' is no longer a dependency
           `src/cbang/FileLocation.h' is no longer a dependency
           `src/cbang/SmartPointer.h' is no longer a dependency
           `src/cbang/Throw.h' is no longer a dependency
           `src/cbang/debug/StackTrace.h' is no longer a dependency
           `src/cbang/StdTypes.h' is no longer a dependency
           `src/cbang/RefCounter.h' is no longer a dependency
           `src/cbang/SStream.h' is no longer a dependency
           `src/cbang/debug/StackFrame.h' is no longer a dependency
           `src/cbang/Deallocators.h' is no longer a dependency
           `src/cbang/util/FormatCheck.h' is no longer a dependency

Regards,
Sergey

[sergey-239]

Joseph,

it looks like the list of dependencies includes whatever is in the supplied source directory as you use simple directory scanner in defining resources builder. I understand that this does not influences outcome, the worst it could do is just trigger unnecessary re-builds when something get touched in the source directory, that is not my case anyway.
As I mentioned already, I have the workaround so this is not an urgent thing to fix.
At least this issue enforced me to start study scons: I have never seen it before. To be honest, I like scons more and more and as a side effect I will improve my python skills by exploring scons ;)

Regards,
Sergey