- sql.py: convert setCategoryDescription() INSERT loop to executemany()
(only remaining INSERT-in-loop; rest of sql.py already used executemany)
- spm/plugins/skel.py: add from __future__ import annotations and type
annotations to all 25 abstract interface methods (return types, param types)
using Python 3.10+ union syntax (X | None instead of Optional[X])
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Simplify const_isfileobj(), const_isnumber(), const_debug_write()
and remove unreachable code in rigo/rigo/utils.py::prepare_markup().
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Fix counter_tick(): portage 3.x removed the root/mycpv arguments,
now a zero-argument call returning the new counter value directly
- Add IDEPEND (EAPI 8 install-time deps) to xpak_entries, extraction
dict, _calculate_dependencies and dep_keys (mapped to bdepend_id)
- Add GPKG binary package format support (.gpkg.tar) alongside xpak:
extract_package_metadata() detects format via get_binpkg_format() and
uses PortageGPKG.unpack_metadata()/decompress() for gpkg packages;
append_metadata_to_package() uses PortageGPKG.update_metadata() for gpkg
- Add "gpkg.tar" to binary_packages_extensions()
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Replace deprecated Thread.isAlive() with is_alive() (Python 3.12)
- Fix invalid escape sequences in regex strings (SyntaxWarning → SyntaxError)
- Replace removed unittest aliases: assertEquals→assertEqual,
assertNotEquals→assertNotEqual, assert_→assertTrue,
assertRaisesRegexp→assertRaisesRegex (Python 3.12)
- Replace portage's removed clear_caches() with flush_cache()
- Fix locks.py: reset TLS state on exception in _file_lock_create
to prevent stale shared-lock state poisoning subsequent acquisitions
- Skip os.chown() when gid is None or caller lacks permission
- Fix FastRSS attribute ordering for consistent minidom re-parse behavior
- Update test expected XML to match Python 3.8+ minidom attribute order
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
FakeOutFile and LogFile need .buffer like Python 3 "text" file objects
which is needed when in rigo standard output/error is replaced, and then
it reaches Portage which does this:
if sys.hexversion >= 0x3000000 and fd in (sys.stdout, sys.stderr):
fd = fd.buffer
fd.write(mystr)
(/usr/lib64/python3.6/site-packages/portage/util/__init__.py).
Entropy internal code did not need this.
Note, after this commit, changes done previously:
1)
commit 0869912ec4
[entropy.spm] Rigo related Python 3 fix
File "/usr/lib64/python3.6/site-packages/entropy/spm/plugins/interfaces/portage_plugin/__init__.py", line 101, in _pusher
self._std.buffer.write(chunk)
AttributeError: 'FakeOutFile' object has no attribute 'buffer'
2)
commit 8700aade27
[entropy.spm, rigo] Rigo related Python 3 fixes
1)
File "/usr/lib/python-exec/python3.6/RigoDaemon_app.py", line 362, in _pusher
fobj.write(chunk)
TypeError: write() argument must be str, not bytes
2)
File "/usr/lib64/python3.6/site-packages/entropy/spm/plugins/interfaces/portage_plugin/__init__.py", line 77, in __init__
self.buffer = Writer(self, self._std.buffer)
AttributeError: 'FakeOutFile' object has no attribute 'buffer'
(...)
could be likely reverted; not tested, it could be better to have them anyway to
avoid futher re/encoding/checks if conversions are needed (subjective).
Fixes bug 5899.
File "/usr/lib64/python3.6/site-packages/entropy/spm/plugins/interfaces/portage_plugin/__init__.py", line 101, in _pusher
self._std.buffer.write(chunk)
AttributeError: 'FakeOutFile' object has no attribute 'buffer'
1)
File "/usr/lib/python-exec/python3.6/RigoDaemon_app.py", line 362, in _pusher
fobj.write(chunk)
TypeError: write() argument must be str, not bytes
2)
File "/usr/lib64/python3.6/site-packages/entropy/spm/plugins/interfaces/portage_plugin/__init__.py", line 77, in __init__
self.buffer = Writer(self, self._std.buffer)
AttributeError: 'FakeOutFile' object has no attribute 'buffer'
With Python 3, data in sqlite from packagesignatures.gpg column was binary on
read (because it was binary on write) but a string was actually required.
The problem was visible with GPG signed packages: when injected into database
using eit with Python 3, it would then explode during equo update/install on
client side.
error from Python 2:
File "/usr/lib64/python2.7/site-packages/entropy/client/interfaces/package/actions/fetch.py", line 1063, in do_signatures_validation
if hash_val in signatures:
TypeError: writable buffers are not hashable
error from Python 3:
File "/usr/lib64/python3.6/site-packages/entropy/client/interfaces/package/actions/fetch.py", line 1023, in do_compare_gpg
tmp_f.write(hash_val)
TypeError: write() argument must be str, not bytes
Added a new way to retrieve list of kernel available
that uses directory /etc/kernels/availables/{sabayon,sabayon-lts}.
If this directory is present is not used current logic for
retrieve the list of kernel.
Under /etc/kernels/avaiables/ directory files with 'lts' extension
are used for retrieve list of LTS kernels.
An user could return his kernel list just if he creates and
maintains his file in a similar format:
$# cat /etc/kernel/availables/mykernel
sys-kernel/linux-sabayon:4.9
sys-kernel/linux-mykernel:4.4
This logic now is used as an additional feature after retrieve
the existing kernels based on virtual/linux-binary or
virtual/linux-binary-lts.
ConfigurationFiles._load_maybe_add (equo.client.misc): path is in bytes here
_std_write (entropy.output): fixes crash in equo conf update with "Replace
original with update" when there is an error in which case msg can be bytes:
File ".../entropy/lib/entropy/tools.py", line 687, in movefile
print_generic("!!! Failed to move", src, "to", dest)
File ".../entropy/lib/entropy/output.py", line 596, in print_generic
_std_write(msg, stderr = stderr)
File ".../entropy/lib/entropy/output.py", line 493, in _std_write
obj.write(msg)
Function callers, in order (during eit commit):
1. filename='lib/entropy/server/interfaces/main.py', function='switch_default_repository', code_context=[' self.close_repositories()\n'], index=0)
2. filename='server/eit/commands/command.py', function='_call_exclusive', code_context=[' server.close_repositories()\n'], index=0)
3. filename='lib/entropy/client/interfaces/client.py', function='destroy', code_context=[' self.close_repositories(mask_clear = False)\n'], index=0)
4. filename='lib/entropy/server/interfaces/main.py', function='destroy', code_context=[' self.close_repositories()\n'], index=0)
Third one triggers packages set synchronization (which would cause "dictionary
changed size during iteration"), and marks sets as being synchronized.
Fourth does not trigger sets synchronization, so closes all repositories
without new ones being opened in the meantime.
Side note:
It's similar to this in "client," lib/entropy/client/interfaces/methods.py:
def close_repositories(self, mask_clear = True):
...
# list() -> python3 support
for item, val in list(repo_cache.items()):
...
repo_cache.pop(item).close(_token = repository_id)
...
but (based on shallow look), it doesn't do as much magic; just calls .pop()
(not sure if there are similar side effects in close() there, though), so the
explanation may possibly apply to the lib/entropy/server function only.
1. Command to execute (diff) was getting arguments like b"path".
2. Crash when priting message.
3. In Python 3, b'x'[0] is int, not str, so a test (if) was (silently)
always true. This way, upon eit commit (at least) and with Python 3,
warning about not merged configuration files was newer printed.
(3) prevented prior discovery of (1) and (2).
Based on my checks, there are no more problems like (3), but I cannot
tell for sure.
[Not tagged so in their git repository... whatever.]
With -R / it prints bogus lines making Entropy crash:
File "/usr/lib64/python2.7/site-packages/portage/dbapi/vartree.py", line 748, in aux_get
raise KeyError(mycpv)
KeyError: 'plib_registry:'
The change was done in portage-utils, commit f05c78008b1754a79e31e793a67d07ed8f5d11bc
Make qfile also check the prune lib registry
and the problem with qfile was reported to Gentoo in bug 699558.
It will break with older qfile but Entropy should fallback to slower,
non-qfile code branch.
In 0.74 -e was available, and in 0.80 it is not present.
In both versions, and with the same set of options used, -e and -v
provide the same result so -v is now used to work on both.
Changed in portage-utils, commit 951a8711a59b1a7d49125f5f5214ff1ae9e50074:
qfile: drop non-functional --exact option
Bug: https://bugs.gentoo.org/678632
The module entropy_path_loader (used only for running from within the
checkout; otherwise not even installed) is made to provide the _entropy
namespace.
(Other ideas instead of this entropy_path_loader change would be to
reorganise files layout; drop support for running from the checkout as
is - and perhaps require virtualenvs; require sourcing a script that
sets PYTHONPATH. However, as implemented, it is not intrusive, and the
good part is that it is quite isolated, not used in normal usage after
installation. Basically, it only does sys.path + provides _entropy
namespace.)
The idea is that:
- entropy.* imports will work as before (so any 3rd party clients will
work as always) - installed in "entropy" package,
- new "_entropy" package to hold a namespace for private modules (like
ones that required adding special directories to sys.path).
(Underscored name for a top level Python module is not very common...
anyway, it was inspired by "_emerge.")
Layout:
site-packages/
entropy (backwards compatible)
const.py
...
kswitch (also toplevel to keep compatibility)
...
_entropy
eit
magneto
matter
rigo
RigoDaemon
solo
(Note that site-packages does not need to be actually Python's
site-packages directory but anything as it is controlled by an argument
to make. It is however intended to be the sitedir.)
Another idea for a layout would be one that mimics sources checkout, but
the layout there is somewhat scattered. (And some ugliness would be
needed to make them modules before implicit namespaces from Python 3.3.
Anyway, imports would be long and ugly.)
Now, the layout of installed Entropy is lean; installation to virtualenv
is also possible (though there would be a need to call scripts like
"python equo.py" as shebangs are not converted).
Follow up changes are needed to make it work.
After this commit alone it would not work when installed (unless module
paths are set in a special way). Next changes will introduce
installation to site-packages so no custom PYTHONPATH will be necessary.
pysqlite has a bug on use VACUUM with py3.6 (see https://github.com/ghaering/pysqlite/issues/109)
Hereinafter, exception related to eit push --quick --force <REPO> command:
Traceback (most recent call last):
File "/usr/lib/entropy/lib/entropy/db/sql.py", line 166, in _proxy_call
return method(*args, **kwargs)
sqlite3.OperationalError: cannot VACUUM from within a transaction
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/bin/eit", line 17, in <module>
main()
File "/usr/lib/entropy/server/eit/main.py", line 114, in main
exit_st = func(*func_args)
File "/usr/lib/entropy/server/eit/commands/command.py", line 237, in _call_exclusive
return func(server)
File "/usr/lib/entropy/server/eit/commands/push.py", line 172, in _push
rc = self._push_repo(entropy_server, repository_id)
File "/usr/lib/entropy/server/eit/commands/push.py", line 184, in _push_repo
rc = self.__push_repo(entropy_server, repository_id)
File "/usr/lib/entropy/server/eit/commands/push.py", line 309, in __push_repo
sts = self.__sync_repo(entropy_server, repository_id)
File "/usr/lib/entropy/server/eit/commands/push.py", line 262, in __sync_repo
enable_download = False, force = self._force)
File "/usr/lib/entropy/lib/entropy/server/interfaces/mirrors.py", line 1673, in sync_repository
enable_upload, enable_download, force = force)
File "/usr/lib/entropy/lib/entropy/server/interfaces/db.py", line 231, in update
enable_upload, enable_download, force = force).update()
File "/usr/lib/entropy/lib/entropy/server/interfaces/db.py", line 404, in update
rc, fine_uris, broken_uris = self._sync()
File "/usr/lib/entropy/lib/entropy/server/interfaces/db.py", line 1873, in _sync
broken_uris = self._upload(uris)
File "/usr/lib/entropy/lib/entropy/server/interfaces/db.py", line 1529, in _upload
self._shrink_and_close(dbconn)
File "/usr/lib/entropy/lib/entropy/server/interfaces/db.py", line 1234, in _shrink_and_close
entropy_repository.vacuum()
File "/usr/lib/entropy/lib/entropy/db/sqlite.py", line 703, in vacuum
self._cursor().execute("vacuum")
File "/usr/lib/entropy/lib/entropy/db/sqlite.py", line 58, in execute
cur = self._proxy_call(self._cur.execute, *args, **kwargs)
File "/usr/lib/entropy/lib/entropy/db/sql.py", line 173, in _proxy_call
raise OperationalError(err)
entropy.db.exceptions.OperationalError: cannot VACUUM from within a transaction
