This makes possible to avoid directory hot spots on repository mirrors.
This commit requires some mileage and real-world testing, but it
seems to be running good on a relatively small repository.
No backward compatibility issues have been reported.
Now that the repository lock is reentrant, it's good to have the
methods take into account direct mode as well. In direct mode,
we explicitly don't want to deal with any kind of locking, because
we accept to manipulate stale data. In order to hide locking code
from the outside and have it transparently managed inside entropy.*
methods, we must respect requests made in direct mode.
The new EntropySQLiteRepository uses ResourceLock, and gains support
for reentrancy, anti-deadlock safety measures (only for nested calls),
unification of memory and file repositories locking code (the semantics
was already the same).
In latency sensitive code paths, the performance penality caused
by file lock contention and memory cache invalidation is too high.
This problem happens in Rigo, which is extremely latency sensitive.
Since we don't want to crap on the user, a way to solve this is
letting API consumers skip the memory cache and read data directly
from the database store. The trade off is that data may be stale,
incomplete, or invalid, but as long as the consumer is aware of this,
that's fine.
Firstly, rwsem is semantically different from flock (but this was known) and
this may confuse the API consumer. Secondly, the locking infrastructure is
purely meant for inter-process synchronization, threads synchronization is
not a current use case.
The old "dependencies" metadata is deprecated. It was found that
the generated metadata might get corrupted by colliding atom strings.
The new implementation avoids collisions completely and is more
efficient.
This commit partially reverts commit d6b7a21314.
Package revision is no longer incremented across package moves.
This has two advantages:
- much less traffic generated on the mirror infrastructure
- less updates for sabayon-limbo users when packages are moved to main
Bumping the revision was required by sabayon-weekly, which had the problem
that some package files were replaced during normal activity on
sabayonlinux.org and sabayon-limbo on the mirror. This generated checksum
errors, thus adding the checksum in the package file name ensures that
Entropy Server will never overwrite package file names unless the checksum
also matches.
Having a SHA1 checksum in the file name is also good for security, and
we may even want to create a SHA1 from the GPG signature in future.
Entropy Server now supports repositories defined inside
/etc/entropy/repositories.conf.d/ files, written using the
syntax detailed below. This improves the ability to enable, disable,
add and remove repositories programmatically. Furthermore, it
makes possible to extend the supported parameters without breaking
backward compatibility.
In order to differentiate Entropy Client repository definitions between
Entropy Server ones, each repository section must start with "[server=".
This is an example of the syntax (with a complete listing
of the supported arguments):
[server=sabayon-limbo]
desc = Sabayon Linux Official Testing Repository
repo = ssh://username@full.host:~username/sabayon-limbo
enabled = <true/false>
[server=sabayon-limbo]
desc = This statement will be ignored.
repo-only = ssh://username@repo.host:~username/sabayon-limbo
pkg-only = ssh://username@pkg.host:~username/sabayon-limbo
[server=sabayon-base]
desc = This is the base repository.
repo-only = ssh://username@repo.host:~username/sabayon-base
pkg-only = ssh://username@pkg.host:~username/sabayon-base
base = <true/false>
As you can see, multiple statements for the same repository
are allowed. However, only the first desc = statement will be
considered, while there can be as many {pkg,repo}* = as you want.
The repository order is important, but this is guaranteed by the
fact that configuration files are parsed in lexical order.
Statements description:
- "desc": stands for description, the repository name description.
- "repo": the push & pull URI, for both packages and repository database.
- "repo-only": same as repo, but only for the repository database
push & pull.
- "pkg-only": same as repo, but only for the packages push & pull.
The supported protocols are those supported by entropy.fetchers.
- "enabled": if set, its value can be either "true" or "false". The default
value is "true". It indicates if a repository is configured
but currently disabled or enabled. Please take into account
that config files in /etc/entropy/repositories.conf.d/ starting
with "_" are considered to contain disabled repositories. This
is just provided for convienence.
- "base": if set, its value can be either "true" or "false". The default
value is "false". If no repository has the flag set, the first
listed repository will be the base one. Only the first repository
with "base = true" will be considered. The base repository is the
repository that is considered base for all the others
(the main one).
