Discussion:
[Freeipa-devel] [freeipa PR#397][opened] Improve wheel building and provide ipaserver wheel for local testing
tiran
2017-01-17 11:44:52 UTC
Permalink
URL: https://github.com/freeipa/freeipa/pull/397
Author: tiran
Title: #397: Improve wheel building and provide ipaserver wheel for local testing
Action: opened

PR body:
"""
The PR improve wheel bundle building and allows ipaserver bundles for local testing
with instrumented build of Python. Debug builds and instrumented builds can have a different binary interface (ABI). For example it is useful for dtrace or test installations in a virtual env. ipaplatform and ipaserver will not be uploaded to PyPI, though.
"""

To pull the PR as Git branch:
git remote add ghfreeipa https://github.com/freeipa/freeipa
git fetch ghfreeipa pull/397/head:pr397
git checkout pr397
tiran
2017-01-23 13:34:38 UTC
Permalink
URL: https://github.com/freeipa/freeipa/pull/397
Author: tiran
Title: #397: Improve wheel building and provide ipaserver wheel for local testing
Action: synchronized

To pull the PR as Git branch:
git remote add ghfreeipa https://github.com/freeipa/freeipa
git fetch ghfreeipa pull/397/head:pr397
git checkout pr397
pvoborni
2017-02-14 16:58:04 UTC
Permalink
URL: https://github.com/freeipa/freeipa/pull/397
Title: #397: Improve wheel building and provide ipaserver wheel for local testing

pvoborni commented:
"""
@tiran I have very vague idea how this is helpful. You have mentioned it during post-devconf "API meeting". But I no longer remember it and description of this PR is very general.

In order to move all the pypi patches forward, we need to document(maybe design) the whole pypi workflow. This is not mentioned in http://www.freeipa.org/page/V4/Build_system_refactoring nor in http://www.freeipa.org/page/V4/Integration_Improvements I.e. how FreeIPA project will work/supply packages to PYPI and what are actually the requirements for these packages. What is expected to work and what not (like everything related to pyhbac).

Right now I have no idea what are the missing blocker parts and what are just nice-to-have things.

Also I don't really like the part that the patches use custom repo of python-nss. But I'm glad that you are working with @jdennis to improve it. @stlaz, with PR #367 what are the remaining usages of python-nss? Could we actually get rid of python-nss completely?



"""

See the full comment at https://github.com/freeipa/freeipa/pull/397#issuecomment-279767185
stlaz
2017-02-15 08:58:16 UTC
Permalink
URL: https://github.com/freeipa/freeipa/pull/397
Title: #397: Improve wheel building and provide ipaserver wheel for local testing

stlaz commented:
"""
@pvoborni The remaining usages are server/CA certificates verification in `certdb.py` and and apparently some encryption/decryption actions in the Vault plugin. @HonzaCholasta already has patches for the former and getting rid of the latter should not be that hard as well.
"""

See the full comment at https://github.com/freeipa/freeipa/pull/397#issuecomment-279953314
stlaz
2017-02-15 08:58:32 UTC
Permalink
URL: https://github.com/freeipa/freeipa/pull/397
Title: #397: Improve wheel building and provide ipaserver wheel for local testing

stlaz commented:
"""
@pvoborni The remaining usages are server/CA certificates verification in `certdb.py` and and apparently some encryption/decryption actions in the Vault plugin. @HonzaCholasta already has patches for the former and getting rid of the latter should not be that hard as well.
"""

See the full comment at https://github.com/freeipa/freeipa/pull/397#issuecomment-279953314
tiran
2017-02-21 09:42:59 UTC
Permalink
URL: https://github.com/freeipa/freeipa/pull/397
Title: #397: Improve wheel building and provide ipaserver wheel for local testing

tiran commented:
"""
@pvoborni The main reason for this PR is explained in the initial PR message. I like to run an IPA framework server with specially instrumented Python builds for profiling or for debugging. The special builds are powerful and incredible useful tools to find bugs or hot spots.

Profile and debug builds have a different ABI than standard builds. Therefore I have to compile all C extensions myself to make them compatible with the new ABI. It is much easier than it sounds, because distutils, setuptools and pip just take care of all the complicated bits and pieces. But this works only for native Python packages. SSSD uses its own build system and has no packages on PyPI. It would take too much time and effort to change SSSD now. Commits 1f195bb4 and c69c30c2 make pyhbac and other SSSD components optional.

Commit 905118a1 allows me to build all ipaserver wheel and full ipaclient wheels with ```install``` subpackage for local testing. These packages are not meant to be uploaded to PyPI. They are really just for local testing.

Last but not least 5710587f is a workaround for a python-nss packaging issue. @jdennis is aware of the problem and will address it in due time. We can't get rid of python-nss. Dogtag PKI's Python modules depend on python-nss.
"""

See the full comment at https://github.com/freeipa/freeipa/pull/397#issuecomment-281293368
tiran
2017-02-21 09:47:04 UTC
Permalink
URL: https://github.com/freeipa/freeipa/pull/397
Title: #397: Improve wheel building and provide ipaserver wheel for local testing

tiran commented:
"""
To clarify and emphasis, this PR has nothing to do with the PyPI packaging effort. Zero. Zip. Nada. Nilch!

The sole intent of this PR is debugging and profiling. It gives us tools to find bugs, to increase performance and to reduce memory usage.
"""

See the full comment at https://github.com/freeipa/freeipa/pull/397#issuecomment-281294344
HonzaCholasta
2017-02-21 12:34:11 UTC
Permalink
URL: https://github.com/freeipa/freeipa/pull/397
Title: #397: Improve wheel building and provide ipaserver wheel for local testing

HonzaCholasta commented:
"""
I can't say I agree with this approach. If this is just for testing, surely you can work around the missing `pyhbac` in some isolated spot rather than make the import optional all over the place, when it is in fact required. Maybe inject an empty module into `sys.modules` if it's missing? Or reach out to the SSSD guys and help them add `pyhbac` to PyPI?
"""

See the full comment at https://github.com/freeipa/freeipa/pull/397#issuecomment-281332519
tiran
2017-02-21 16:13:23 UTC
Permalink
URL: https://github.com/freeipa/freeipa/pull/397
Author: tiran
Title: #397: Improve wheel building and provide ipaserver wheel for local testing
Action: synchronized

To pull the PR as Git branch:
git remote add ghfreeipa https://github.com/freeipa/freeipa
git fetch ghfreeipa pull/397/head:pr397
git checkout pr397
tiran
2017-02-22 08:40:20 UTC
Permalink
URL: https://github.com/freeipa/freeipa/pull/397
Title: #397: Improve wheel building and provide ipaserver wheel for local testing

tiran commented:
"""
@HonzaCholasta FreeIPA has conditional imports for SSSD modules in several places, e.g. in the trust plugin. 96f614e closes the gap and applies the same technique to the last unconditional import from SSSD.
"""

See the full comment at https://github.com/freeipa/freeipa/pull/397#issuecomment-281604493
HonzaCholasta
2017-02-22 09:16:44 UTC
Permalink
URL: https://github.com/freeipa/freeipa/pull/397
Title: #397: Improve wheel building and provide ipaserver wheel for local testing

HonzaCholasta commented:
"""
The trust plugin and other trust bits are optional. The cert plugin, which depends on `pyhbac`, is *not* optional, so you can't apply the same logic to it.

An acceptable compromise would be to skip the cert plugin entirely if `pyhbac` is not available:
```python
try:
import pyhbac
except ImportError:
raise errors.SkipPluginModule(reason=_('pyhbac is not installed'))
```
"""

See the full comment at https://github.com/freeipa/freeipa/pull/397#issuecomment-281612799
tiran
2017-02-23 08:51:50 UTC
Permalink
URL: https://github.com/freeipa/freeipa/pull/397
Title: #397: Improve wheel building and provide ipaserver wheel for local testing

tiran commented:
"""
I didn't know about the ```SkipPluginModule``` feature. I agree with you, your solution is more elegant.
"""

See the full comment at https://github.com/freeipa/freeipa/pull/397#issuecomment-281932095
tiran
2017-02-23 09:49:32 UTC
Permalink
URL: https://github.com/freeipa/freeipa/pull/397
Author: tiran
Title: #397: Improve wheel building and provide ipaserver wheel for local testing
Action: synchronized

To pull the PR as Git branch:
git remote add ghfreeipa https://github.com/freeipa/freeipa
git fetch ghfreeipa pull/397/head:pr397
git checkout pr397
tiran
2017-03-01 08:18:21 UTC
Permalink
URL: https://github.com/freeipa/freeipa/pull/397
Author: tiran
Title: #397: Improve wheel building and provide ipaserver wheel for local testing
Action: synchronized

To pull the PR as Git branch:
git remote add ghfreeipa https://github.com/freeipa/freeipa
git fetch ghfreeipa pull/397/head:pr397
git checkout pr397
tiran
2017-03-01 08:18:53 UTC
Permalink
URL: https://github.com/freeipa/freeipa/pull/397
Title: #397: Improve wheel building and provide ipaserver wheel for local testing

tiran commented:
"""
@jdennis released python-nss 1.0.1. I removed my workaround.
"""

See the full comment at https://github.com/freeipa/freeipa/pull/397#issuecomment-283274521
MartinBasti
2017-03-08 14:05:26 UTC
Permalink
URL: https://github.com/freeipa/freeipa/pull/397
Title: #397: Improve wheel building and provide ipaserver wheel for local testing

MartinBasti commented:
"""
needs rebase
"""

See the full comment at https://github.com/freeipa/freeipa/pull/397#issuecomment-285048889
tiran
2017-03-14 23:36:00 UTC
Permalink
URL: https://github.com/freeipa/freeipa/pull/397
Author: tiran
Title: #397: Improve wheel building and provide ipaserver wheel for local testing
Action: synchronized

To pull the PR as Git branch:
git remote add ghfreeipa https://github.com/freeipa/freeipa
git fetch ghfreeipa pull/397/head:pr397
git checkout pr397
tiran
2017-03-14 23:36:43 UTC
Permalink
URL: https://github.com/freeipa/freeipa/pull/397
Title: #397: Improve wheel building and provide ipaserver wheel for local testing

tiran commented:
"""
I've moved the code to cert.py and raise SkipPluginModule from there.
"""

See the full comment at https://github.com/freeipa/freeipa/pull/397#issuecomment-286595688
tiran
2017-03-16 06:45:24 UTC
Permalink
URL: https://github.com/freeipa/freeipa/pull/397
Author: tiran
Title: #397: Improve wheel building and provide ipaserver wheel for local testing
Action: synchronized

To pull the PR as Git branch:
git remote add ghfreeipa https://github.com/freeipa/freeipa
git fetch ghfreeipa pull/397/head:pr397
git checkout pr397
tiran
2017-03-22 10:29:59 UTC
Permalink
URL: https://github.com/freeipa/freeipa/pull/397
Title: #397: Improve wheel building and provide ipaserver wheel for local testing

tiran commented:
"""
PR is blocked by #613
"""

See the full comment at https://github.com/freeipa/freeipa/pull/397#issuecomment-288357734
MartinBasti
2017-03-30 15:01:07 UTC
Permalink
URL: https://github.com/freeipa/freeipa/pull/397
Title: #397: Improve wheel building and provide ipaserver wheel for local testing

MartinBasti commented:
"""
Please rebase, it is ok to me, I see potential for future server unit testing. I will test when rebased. If somebody is against this please say it now.
"""

See the full comment at https://github.com/freeipa/freeipa/pull/397#issuecomment-290438690
tiran
2017-03-31 06:50:59 UTC
Permalink
URL: https://github.com/freeipa/freeipa/pull/397
Author: tiran
Title: #397: Improve wheel building and provide ipaserver wheel for local testing
Action: synchronized

To pull the PR as Git branch:
git remote add ghfreeipa https://github.com/freeipa/freeipa
git fetch ghfreeipa pull/397/head:pr397
git checkout pr397
HonzaCholasta
2017-03-31 06:54:09 UTC
Permalink
URL: https://github.com/freeipa/freeipa/pull/397
Title: #397: Improve wheel building and provide ipaserver wheel for local testing

HonzaCholasta commented:
"""
LGTM.
"""

See the full comment at https://github.com/freeipa/freeipa/pull/397#issuecomment-290632228
tiran
2017-03-31 06:57:51 UTC
Permalink
URL: https://github.com/freeipa/freeipa/pull/397
Title: #397: Improve wheel building and provide ipaserver wheel for local testing

tiran commented:
"""
Thanks @MartinBasti

I rebased the PR and added a small workaround for ```dbus-python```. The package uses make to compile some of its internal dependencies. It looks like there is a bug in ```dbus-python```'s makefile. It sometimes fails to compile with my ```MAKEFLAGS=-j4``` env var. ```Makefile.am``` line 253 sets MAKEFLAGS to empty value for ```pip wheel```.
"""

See the full comment at https://github.com/freeipa/freeipa/pull/397#issuecomment-290632826
MartinBasti
2017-03-31 10:53:03 UTC
Permalink
URL: https://github.com/freeipa/freeipa/pull/397
Title: #397: Improve wheel building and provide ipaserver wheel for local testing

MartinBasti commented:
"""
Build failed:

```
make wheel_bundle IPA_SERVER_WHEELS=1
...
checking for DBUS... no
configure: error: Package requirements (dbus-1 >= 1.6) were not met:

No package 'dbus-1' found

Consider adjusting the PKG_CONFIG_PATH environment variable if you
installed software in a non-standard prefix.

Alternatively, you may set the environment variables DBUS_CFLAGS
and DBUS_LIBS to avoid the need to call pkg-config.
See the pkg-config man page for more details.
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-build-l97uxR/dbus-python/setup.py", line 106, in <module>
'build_ext': BuildExt,
File "/usr/lib64/python2.7/distutils/core.py", line 151, in setup
dist.run_commands()
File "/usr/lib64/python2.7/distutils/dist.py", line 953, in run_commands
self.run_command(cmd)
File "/usr/lib64/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/usr/lib/python2.7/site-packages/wheel/bdist_wheel.py", line 199, in run
self.run_command('build')
File "/usr/lib64/python2.7/distutils/cmd.py", line 326, in run_command
self.distribution.run_command(command)
File "/usr/lib64/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/tmp/pip-build-l97uxR/dbus-python/setup.py", line 62, in run
cwd=builddir)
File "/usr/lib64/python2.7/subprocess.py", line 186, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/tmp/pip-build-l97uxR/dbus-python/configure', '--disable-maintainer-mode', 'PYTHON=/usr/bin/python', '--prefix=/tmp/pip-build-l97uxR/dbus-python/build/temp.linux-x86_64-2.7/prefix']' returned non-zero exit status 1

----------------------------------------
Failed building wheel for dbus-python
Running setup.py clean for dbus-python
Running setup.py bdist_wheel for MarkupSafe ... done
Stored in directory: /tmp/freeipa/dist/bundle
Running setup.py bdist_wheel for pycparser ... done
Stored in directory: /tmp/freeipa/dist/bundle
Running setup.py bdist_wheel for configparser ... done
Stored in directory: /tmp/freeipa/dist/bundle
Successfully built cryptography python-yubico pyusb python-nss pyldap netifaces gssapi MarkupSafe pycparser configparser
Failed to build dbus-python
ERROR: Failed to build one or more wheels
Makefile:1222: recipe for target 'wheel_bundle' failed

```
"""

See the full comment at https://github.com/freeipa/freeipa/pull/397#issuecomment-290682068
tiran
2017-03-31 11:34:59 UTC
Permalink
URL: https://github.com/freeipa/freeipa/pull/397
Title: #397: Improve wheel building and provide ipaserver wheel for local testing

tiran commented:
"""
You need dbus-devel package.

I opened https://pagure.io/freeipa/issue/6842 to track lack of documentation.
"""

See the full comment at https://github.com/freeipa/freeipa/pull/397#issuecomment-290689299
MartinBasti
2017-03-31 11:48:00 UTC
Permalink
URL: https://github.com/freeipa/freeipa/pull/397
Title: #397: Improve wheel building and provide ipaserver wheel for local testing

MartinBasti commented:
"""
And document in `BUILD.txt` how to build wheels
"""

See the full comment at https://github.com/freeipa/freeipa/pull/397#issuecomment-290691545
MartinBasti
2017-03-31 11:47:30 UTC
Permalink
URL: https://github.com/freeipa/freeipa/pull/397
Title: #397: Improve wheel building and provide ipaserver wheel for local testing

MartinBasti commented:
"""
So put it into specfile to `with_wheels` section
"""

See the full comment at https://github.com/freeipa/freeipa/pull/397#issuecomment-290691425
tiran
2017-03-31 13:28:06 UTC
Permalink
URL: https://github.com/freeipa/freeipa/pull/397
Author: tiran
Title: #397: Improve wheel building and provide ipaserver wheel for local testing
Action: synchronized

To pull the PR as Git branch:
git remote add ghfreeipa https://github.com/freeipa/freeipa
git fetch ghfreeipa pull/397/head:pr397
git checkout pr397
tiran
2017-03-31 14:29:17 UTC
Permalink
URL: https://github.com/freeipa/freeipa/pull/397
Title: #397: Improve wheel building and provide ipaserver wheel for local testing

tiran commented:
"""
@MartinBasti ```dbus-devel``` is in the ```with_wheels``` section. Documentation is part of https://pagure.io/freeipa/issue/6842 .
"""

See the full comment at https://github.com/freeipa/freeipa/pull/397#issuecomment-290727605
MartinBasti
2017-04-03 07:35:34 UTC
Permalink
URL: https://github.com/freeipa/freeipa/pull/397
Title: #397: Improve wheel building and provide ipaserver wheel for local testing

MartinBasti commented:
"""
@tiran sorry, but then something doesn't work as expected

```
$ dnf builddep -b -D "with_wheels 1" --spec freeipa.spec.in
<everything already installed>

$ make wheel_bundle IPA_SERVER_WHEELS=1
...
Failed building wheel for dbus-python

$ git grep -in dbus-devel freeipa.spec.in
freeipa.spec.in:149:BuildRequires: dbus-devel

$ rpm -q dbus-devel
package dbus-devel is not installed
```

Probably because `%global with_wheels 0` is defined in the spec file, so it always overrides my CLI settings
"""

See the full comment at https://github.com/freeipa/freeipa/pull/397#issuecomment-291069185
tiran
2017-04-03 07:48:36 UTC
Permalink
URL: https://github.com/freeipa/freeipa/pull/397
Author: tiran
Title: #397: Improve wheel building and provide ipaserver wheel for local testing
Action: synchronized

To pull the PR as Git branch:
git remote add ghfreeipa https://github.com/freeipa/freeipa
git fetch ghfreeipa pull/397/head:pr397
git checkout pr397
tiran
2017-04-03 08:04:13 UTC
Permalink
URL: https://github.com/freeipa/freeipa/pull/397
Author: tiran
Title: #397: Improve wheel building and provide ipaserver wheel for local testing
Action: synchronized

To pull the PR as Git branch:
git remote add ghfreeipa https://github.com/freeipa/freeipa
git fetch ghfreeipa pull/397/head:pr397
git checkout pr397
MartinBasti
2017-04-03 08:28:40 UTC
Permalink
URL: https://github.com/freeipa/freeipa/pull/397
Title: #397: Improve wheel building and provide ipaserver wheel for local testing

MartinBasti commented:
"""
Missing dependency `dbus-glib-devel` otherwise it works
"""

See the full comment at https://github.com/freeipa/freeipa/pull/397#issuecomment-291079545
tiran
2017-04-03 08:41:09 UTC
Permalink
URL: https://github.com/freeipa/freeipa/pull/397
Author: tiran
Title: #397: Improve wheel building and provide ipaserver wheel for local testing
Action: synchronized

To pull the PR as Git branch:
git remote add ghfreeipa https://github.com/freeipa/freeipa
git fetch ghfreeipa pull/397/head:pr397
git checkout pr397
tiran
2017-04-03 09:23:49 UTC
Permalink
URL: https://github.com/freeipa/freeipa/pull/397
Title: #397: Improve wheel building and provide ipaserver wheel for local testing

tiran commented:
"""
Sigh, I got it.
"""

See the full comment at https://github.com/freeipa/freeipa/pull/397#issuecomment-291091639
MartinBasti
2017-04-03 09:50:01 UTC
Permalink
URL: https://github.com/freeipa/freeipa/pull/397
Title: #397: Improve wheel building and provide ipaserver wheel for local testing

MartinBasti commented:
"""
Can you rebase please? I see Ben's commits in this PR
"""

See the full comment at https://github.com/freeipa/freeipa/pull/397#issuecomment-291097531
tiran
2017-04-03 10:20:23 UTC
Permalink
URL: https://github.com/freeipa/freeipa/pull/397
Author: tiran
Title: #397: Improve wheel building and provide ipaserver wheel for local testing
Action: synchronized

To pull the PR as Git branch:
git remote add ghfreeipa https://github.com/freeipa/freeipa
git fetch ghfreeipa pull/397/head:pr397
git checkout pr397
MartinBasti
2017-04-03 11:04:46 UTC
Permalink
URL: https://github.com/freeipa/freeipa/pull/397
Title: #397: Improve wheel building and provide ipaserver wheel for local testing

Label: +ack
MartinBasti
2017-04-03 11:09:09 UTC
Permalink
URL: https://github.com/freeipa/freeipa/pull/397
Title: #397: Improve wheel building and provide ipaserver wheel for local testing

Label: +pushed
MartinBasti
2017-04-03 11:09:14 UTC
Permalink
URL: https://github.com/freeipa/freeipa/pull/397
Title: #397: Improve wheel building and provide ipaserver wheel for local testing

MartinBasti commented:
"""
master:

* 3064b890e24a5056c77f19c3951cfb59d49366f8 Conditionally import pyhbac
* 7c9df35d3dd812543cf537c0762c6db728379919 Add extra_requires for additional dependencies
* ae1c2086db3efb51341025ae25f9b39060868ac1 Add an option to build ipaserver wheels
* 40a60675f3feb118af70562582399afefe97214d Don't hard-code with_wheels


"""

See the full comment at https://github.com/freeipa/freeipa/pull/397#issuecomment-291113162
MartinBasti
2017-04-03 11:09:19 UTC
Permalink
URL: https://github.com/freeipa/freeipa/pull/397
Author: tiran
Title: #397: Improve wheel building and provide ipaserver wheel for local testing
Action: closed

To pull the PR as Git branch:
git remote add ghfreeipa https://github.com/freeipa/freeipa
git fetch ghfreeipa pull/397/head:pr397
git checkout pr397

Loading...