The idea is that:
- entropy.* imports will work as before (so any 3rd party clients will
work as always) - installed in "entropy" package,
- new "_entropy" package to hold a namespace for private modules (like
ones that required adding special directories to sys.path).
(Underscored name for a top level Python module is not very common...
anyway, it was inspired by "_emerge.")
Layout:
site-packages/
entropy (backwards compatible)
const.py
...
kswitch (also toplevel to keep compatibility)
...
_entropy
eit
magneto
matter
rigo
RigoDaemon
solo
(Note that site-packages does not need to be actually Python's
site-packages directory but anything as it is controlled by an argument
to make. It is however intended to be the sitedir.)
Another idea for a layout would be one that mimics sources checkout, but
the layout there is somewhat scattered. (And some ugliness would be
needed to make them modules before implicit namespaces from Python 3.3.
Anyway, imports would be long and ugly.)
Now, the layout of installed Entropy is lean; installation to virtualenv
is also possible (though there would be a need to call scripts like
"python equo.py" as shebangs are not converted).
Follow up changes are needed to make it work.
After this commit alone it would not work when installed (unless module
paths are set in a special way). Next changes will introduce
installation to site-packages so no custom PYTHONPATH will be necessary.
pysqlite has a bug on use VACUUM with py3.6 (see https://github.com/ghaering/pysqlite/issues/109)
Hereinafter, exception related to eit push --quick --force <REPO> command:
Traceback (most recent call last):
File "/usr/lib/entropy/lib/entropy/db/sql.py", line 166, in _proxy_call
return method(*args, **kwargs)
sqlite3.OperationalError: cannot VACUUM from within a transaction
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/bin/eit", line 17, in <module>
main()
File "/usr/lib/entropy/server/eit/main.py", line 114, in main
exit_st = func(*func_args)
File "/usr/lib/entropy/server/eit/commands/command.py", line 237, in _call_exclusive
return func(server)
File "/usr/lib/entropy/server/eit/commands/push.py", line 172, in _push
rc = self._push_repo(entropy_server, repository_id)
File "/usr/lib/entropy/server/eit/commands/push.py", line 184, in _push_repo
rc = self.__push_repo(entropy_server, repository_id)
File "/usr/lib/entropy/server/eit/commands/push.py", line 309, in __push_repo
sts = self.__sync_repo(entropy_server, repository_id)
File "/usr/lib/entropy/server/eit/commands/push.py", line 262, in __sync_repo
enable_download = False, force = self._force)
File "/usr/lib/entropy/lib/entropy/server/interfaces/mirrors.py", line 1673, in sync_repository
enable_upload, enable_download, force = force)
File "/usr/lib/entropy/lib/entropy/server/interfaces/db.py", line 231, in update
enable_upload, enable_download, force = force).update()
File "/usr/lib/entropy/lib/entropy/server/interfaces/db.py", line 404, in update
rc, fine_uris, broken_uris = self._sync()
File "/usr/lib/entropy/lib/entropy/server/interfaces/db.py", line 1873, in _sync
broken_uris = self._upload(uris)
File "/usr/lib/entropy/lib/entropy/server/interfaces/db.py", line 1529, in _upload
self._shrink_and_close(dbconn)
File "/usr/lib/entropy/lib/entropy/server/interfaces/db.py", line 1234, in _shrink_and_close
entropy_repository.vacuum()
File "/usr/lib/entropy/lib/entropy/db/sqlite.py", line 703, in vacuum
self._cursor().execute("vacuum")
File "/usr/lib/entropy/lib/entropy/db/sqlite.py", line 58, in execute
cur = self._proxy_call(self._cur.execute, *args, **kwargs)
File "/usr/lib/entropy/lib/entropy/db/sql.py", line 173, in _proxy_call
raise OperationalError(err)
entropy.db.exceptions.OperationalError: cannot VACUUM from within a transaction
Otherwise "entropy" module loading logic was delayed, excluding this
import (which was done early, before entropy_path_loader logic):
from entropy.locks import SimpleFileLock
and it failed when system wide PYTHONPATH is dropped.
There was a comment not to do an early import due to --no-pid-handling.
The comment was added in:
commit 59e5fd737e
Date: Tue Mar 1 13:13:18 2011 +0100
[magneto] fix lock file handling
but then the logic was reworked in:
commit 5d315c70bd
Date: Sun Feb 24 23:34:49 2013 +0100
[entropy.const] drop /var/run/entropy/entropy.lock file and directory handling
and then passing --no-pid-handling (which was unused at this point) from
mangeto file was removed in:
commit 83b8991c95
Date: Wed Feb 27 20:59:14 2013 +0000
[magneto] rewrite systray icons management, improve KDE, GNOME and MATE integration
This way a proper separation between Python installs can be achieved.
With no PYTHON_SITEDIR, installation paths are exactly the same as
before this change.
In practise, passing nonstandard path will break Entropy but so it is
also with the currently available LIBDIR. This is a concern of a future
improvement (at least the PYTHON_SITEDIR path, not necessarily LIBDIR as
PYTHON_SITEDIR *will* be different if it's installed in Python specific
directories).
This way there is no hardcoded /usr/lib/rigo, and the "executable" part
is in a well known location which can be replicated on system for
different Python implementations. (As in: it will be like /usr/bin/foo.py
which on Gentoo links to a system wrapper making it use configured
Python.)
For convenience (seemingly, and it really is convenient) equo and other
tools can be run from the checkout, and Entropy modules are loaded from
the checkout. Now there is a strict separation when system paths and
when paths from the checkout are used.
It makes it a bit more robust, secure and preditable at the cost of
a little more complexity.
A pleasant side effect of this change is that it is not required to
change directory to the tool (to use non-system one), as paths in the
checkout are relative to scripts.
Imports in lib/tests were not adjusted.
This is a ";" separated data input (even though we may get directory
paths containing ";"... but none so far is expected to be legit.)
Split by ";" first and then deal with spaces in paths.
It looks like Portage now stores unicode paths correctly in its metadata
as opposed to what it used to be. We need to make sure that we parse those
"CONTENTS" file and content metadata in general using the correct encoding.
This will allow us to store and retrieve such metadata from the sqlite3
database correctly and also match the stored paths with the filesystem
paths exactly.
This commit may need a bit more real-life testing. Backward compat
wrt old Entropy and Portage tbz2 files should be as expected.
Unit tests attached.
Historically, Portage + Py2 were not dealing well with file paths that
were containing non-ascii chars. The situation has apparently improved
and we are able to accept paths with the correct encoding in Entropy
(separate patch will come). We need to guard against broken paths anyway,
so let's try to check which one of the paths is the one that has the
most chance to be the right one. Also, with python3 this won't be
a problem anymore...
Due to the migration to real virtual packages (away from PROVIDE= due
to EAPI=7 constraints), we failed to resolve kernel binary packages
correctly, thus packages that require them (external kernel modules.)
This patch fixes it, by making sure that we resolve new virtuals correctly.
We expect new virtuals to list kernel packages in their RDEPEND= (runtime
dependecies.)
Sławomir Nizio
Ettore Di Giacinto
Ettore Di Giacinto
Fabio Erculiani
Geaaru