It has been observed that using os.fdopen() below in multi-threaded
scenarios is causing EBADF (thus OSError). There is probably a racen
condition down in the stack or mkstemp() itself is not guaranteed against
concurrent access. For now, just consume one more file descriptor and
avoid the race completely.
Using subprocess.PIPE and Popen.wait() causes the child process to
block in case of write (stdout) buffer full. Rewrite the whole
function to use simple a simple fd via mkstemp(). This way, even
if the env variable is very long, the process won't hang.
This issue has been observed in dev-texlive/texlive-latexextra-2012's
SRC_URI, which is more than 131k chars long!
If splitdebug was previously enabled for a package, and its
files were installed, when splitdebug is turned off Package kept
printing collision messages about /usr/lib/debug files.
This happened because packages contain /usr/lib/debug paths in
their content metadata (even those with separate splitdebug tarballs)
and this wasn't cleared out before installed packages repository
metadata update. Thus, during the cleanup step, these paths popped out
generating invalid warnings.
It happened that assign_uid_to_installed_package() did not explicitly
invalidate the Portage vdb cache and, while this is used inside
add_installed_package(), which did invalidate the vdb cache on its own
already, it is also called directly by entropy.client.interfaces.package
code.
Moreover, root= value wasn't properly handled throughout the execution
of add_installed_package(), which may have caused cache invalidation
issues described above.
The original idea was to avoid doing cursor and connection resources
cleanup (left by old and dead threads) synchronously every time
_connection() and/or _cursor() is accessed. This strategy also had
a huge drawback: with no activity on the object, resources were
left hanging there forever.
This commit introduces a better strategy for transparent and automatic
cleanup of resources belonging to terminated threads: every time a new
thread_id arrives at _cursor() or _connection(), a new daemon thread
starts and synchronizes with the caller through a simple Thread.join()
(because it's a daemon thread, we can join() daemon threads as well,
even if this is not really compliant with the specs, but it seems to work
just fine in Python).
When the caller thread is joined, it is possible to start the resources
cleanup procedure, carefully taking into account that thread_ids are
recycled and thus there might be clashing with newly created threads.
This helped a design issue to emerge from the sand (like a zombie
at the seaside): it is impossible to cleanup resources left by the
MainThread because this thread never ends living, and if it dies,
everything dies, obviously. So, the first implementation of this new
strategy was NOT touching the MainThread resources but then, the old
behaviour was to kill them as well on EntropyRepository.close().
So, the final version of this patch kept the old buggy behaviour of
touching MainThread stuff (nein, nein, nein, nein would Hitler say).
However, a new keyword argument "safe" has been added to the close()
method so it is possible to start migrating code to the dark side of the
power.
This means nothing really changed for API consumers yet, just entropy.db.sql
code being more efficient (no weird for loops and synchronous crap)
and actually faster (multi-threading ftw).
As explained in the code comments, this is mandatory for scenarios
in where the iterator has to run multiple times because transactions
can be rolled back and replayed indefinitely.
The licensename column is declared as UNIQUE, multiple threads inserting
rows can cause unique constraint violations. Considering the nature of
the data, using "INSERT OR REPLACE" can be considered safe and actually
wanted.
This issue caused a load of issues with the ca-certificates.
Example of partial readline():
0|obj|/usr/share/ca-certificates/mozilla/NetLock_Arany_=Class_Gold=_F\xc3\x85
and the next call:
\xc2\x91tan\xc3\x83\xc2\xbas\xc3\x83\xc2\xadtv\xc3\x83\xc2\xa1ny.crt\n
Trying to workaround it by reading ahead if line does not end with \n
Fabio Erculiani