sablink/for-gentoo
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

[overlay-split] remove Sabayon stuff

Browse Source 
[overlay-split] dev-vcs: remove, split ebuilds

[overlay-split] app-crypt/* remove split ebuilds

[overlay-split] www-apps: remove split stuff

[overlay-split] app-office: remove libreoffice*

[overlay-split] remove stuff

[overlay-split] remove most from www-client

[sys-apps/hal] Removing depricated sys-apps/hal ebuilds.

The sys-apps/hal package has been depricated.  Removing sys-apps/hal
ebuilds from the overlay.

[app-emulation/wine] Removing app-emulation/wine that depend on hal.

Removing the app-emulation/wine ebuilds that depend on hal, since
hal has been depricated.

[overlay-split] remove some stuff from app-misc and their revdeps

[overlay-split] remove some stuff from media-gfx

[overlay-split] remove split (zen-)nmap

[overlay-split] x11-drivers -> remove

[overlay-split] media-libs and media-gfx -> remove some stuff

[overlay-split] x11-apps and net-print done

[overlay-split] dev-util done

json-glib: remove old and unmasked 9999

[overlay-split] dev-python done

remove httlib2 - not needed in the overlays completely
urlgrabber - probably fixed in Portage - not sure, not brought
 by me to the overlay, let's remove from this branch only

[overlay-split] media-plugins, net-dns, x11-wm

remove alsa-plugins (let 'em use Portage's)
remove split avahi
remove customised x11-wm/awesome (enlightenment is left due to short
discussion)

[overlay-split] dev-libs

remove dev-libs/libdbusmenu - this version is not used even
in Entropy, let's remove it from 'gentoo' and leave overlay
users Portage's version
Enlightenment stuff is left here "as always".
d0_blind_id is unmasked 9999, but it's for xonotic and it
doesn't have large rev. dep list.

[overlay-split] media-sound done

remove guayadeque, fixed in Gentoo, removed from master already
remove pulseaudio, let the Gentoo folks use "their"

[overlay-split] x11-themes: remove Sabayon-related stuff

[overlay-split] net-wireless: remove some stuff

acx - stale files only
b43-firmware - old, doesn't seem to add any value currently,
 unlikely useful for Gentoo users
broadcom-sta - same as above
btsco* - old, bad dep, probably doesn't even compile, unused
libertas-firmware - like broad-sta

[overlay-split] app-text: remove stuff

epdf - keywored live ebuild, do not want
pastebunz - hard choice, but seems more natural in the other overlay
 (and it can be moved in needed)
poppler* - ofc rm'd split'd ebuilds

[overlay-split] media-tv: remove some stuff

[overlay-split] sys-boot: remove

[overlay-split] gnome-base: remove some stuff

[overlay-split] media-video: remove some stuff

[overlay-split] net-irc: remove split quassel

[overlay-split] sys-apps: remove Sabayon stuff

[overlay-split] net-mail/mailbase: remove mod.

[overlay-split] net-libs/xulrunner: remove Sabayon mod.

[overlay-split] x11-libs: remove some stuff

xvba-video - it's in Portage; and it depends on
 x11-drivers/ati-userspace, which is not in Portage
xvba-sdk - needed for xvba-video, let's move it too
xcb-util - who the heck needs it here

[overlay-split] net-misc: remove hylafax and networkmanager

hylafax - "[net-misc/hylafax] make it work with our media-libs/tiff
compat package"

[overlay-split] sys-fs: rm udev and mdadm

[overlay-split] sys-libs: rm glibc

[overlay-split] x11-base: remove

[overlay-split] app-admin: rm gnome-system-tools

[overlay-split] sys-kernel and sys-power done

[overlay-split] x11-plugins: remove unmasked live ebuild

[overlay-split] kde-misc: rm unmasked live eb. that would override Portage's

[overlay-split] kde-base: rm all, Sabayon mod.

[overlay-split] complete this phase - rm remaining stuff!
This commit is contained in:
Sławomir Nizio
2011-11-12 18:32:09 +01:00
committed by Sławomir Nizio
parent 5646bef726
commit 1cad745dcf
2006 changed files with 0 additions and 269357 deletions
Download Patch File Download Diff File Expand all files Collapse all files
dev-python/pmw/Manifest
-4
View File
@@ -1,4 +0,0 @@
AUX 1.3.2-python2.5.patch 2030 RMD160 5bb1f44f8d67903fabfba7c51e300e9638bc64db SHA1 ede19deb49c5531462186496b4d47134a8e420bb SHA256 7f0d3940926ca59245251b8c471d446741096fc35cbd4198d2b4ec4985029f35
AUX pmw-1.3.2-install-no-docs.patch 493 RMD160 bb6b8017e19da6627f15bd6d8a0d7ecf1e166faf SHA1 fc1c2f9eef884e74ca0fe8f3b720563fbedcf411 SHA256 4511b359a315fc5c2c5c143622f32c227a620a2f916f94aee7dcea9d1cb95145
DIST Pmw.1.3.2.tar.gz 399262 RMD160 ad3fca73405a8c8a7e2ec81096b00665e2937964 SHA1 e2d8dd6eed7378acf8723c860107a12d7e808227 SHA256 0c9ab8f9cbd57f926e22253d73048fdccfb7e033c3482b083a6fb41848074a18
EBUILD pmw-1.3.2-r2.ebuild 1116 RMD160 27c9470220014a58b0c1f9d0a168d78b95ae59db SHA1 05ea7cc4a898f5b83f3145c08dcf6d39624dfd39 SHA256 bbcc4aff9a4ea3a8daa65e185bf68a9a53c3bd4420fa7b3ffd4b12ee6bf3f58c
dev-python/pmw/files/1.3.2-python2.5.patch
-54
View File
@@ -1,54 +0,0 @@
Source: http://github.com/nanotube/pmw_fixes/commit/5e6dcbdaef2bb6c40037b922dd0efa081f1575ab (Daniel Folkinshteyn)
fix bug in OptionMenu.setitems, introduced with the fixing of this python
bug:
http://bugs.python.org/issue1342811
versions of python 2.5.4 and newer now automatically delete commands when
items are deleted
however, that causes error if the command is deleted manually beforehand.
old versions, on the contrary, require manual command deletion to free up
the objects.
so we put in a python version check to see if we need to manually delete
commands.
fixes the Pmw bug referred to here:
http://sourceforge.net/forum/forum.php?thread_id=3283195&forum_id=33675
and here:
http://sourceforge.net/tracker/?func=detail&aid=2795731&group_id=10743&ati
d=110743
--- a/src/Pmw/Pmw_1_3/lib/PmwOptionMenu.py
+++ b/src/Pmw/Pmw_1_3/lib/PmwOptionMenu.py
@@ -1,6 +1,7 @@
import types
import Tkinter
import Pmw
+import sys
class OptionMenu(Pmw.MegaWidget):
@@ -59,11 +60,17 @@ class OptionMenu(Pmw.MegaWidget):
def setitems(self, items, index = None):
- # Clean up old items and callback commands.
- for oldIndex in range(len(self._itemList)):
- tclCommandName = str(self._menu.entrycget(oldIndex, 'command'))
- if tclCommandName != '':
- self._menu.deletecommand(tclCommandName)
+ # python version check
+ # python versions >= 2.5.4 automatically clean commands
+ # and manually cleaning them causes errors when deleting items
+
+ if sys.version_info[0] * 100 + sys.version_info[1] * 10 + \
+ sys.version_info[2] < 254:
+ # Clean up old items and callback commands.
+ for oldIndex in range(len(self._itemList)):
+ tclCommandName = str(self._menu.entrycget(oldIndex, 'command'))
+ if tclCommandName != '':
+ self._menu.deletecommand(tclCommandName)
self._menu.delete(0, 'end')
self._itemList = list(items)
dev-python/pmw/files/pmw-1.3.2-install-no-docs.patch
-13
View File
@@ -1,13 +0,0 @@
diff -uNr Pmw.1.3.2.orig/src/setup.py Pmw.1.3.2/src/setup.py
--- Pmw.1.3.2.orig/src/setup.py 2008-08-02 15:40:19.000000000 -0430
+++ Pmw.1.3.2/src/setup.py 2008-08-02 15:40:36.000000000 -0430
@@ -16,9 +16,7 @@
'Pmw.Pmw_1_3.lib',],
package_data={'Pmw': ['Pmw_1_3/lib/Pmw.def',
- 'Pmw_1_3/doc/*',
'Pmw_1_3/contrib/*',
- 'Pmw_1_3/demos/*',
'Pmw_1_3/tests/*',
'Pmw_1_3/bin/*',
]
dev-python/pmw/pmw-1.3.2-r2.ebuild
-50
View File
@@ -1,50 +0,0 @@
# Copyright 1999-2010 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: Exp $
EAPI="3"
PYTHON_DEPEND="2"
SUPPORT_PYTHON_ABIS="1"
RESTRICT_PYTHON_ABIS="3.*"
inherit distutils eutils
MY_P="Pmw.${PV}"
DESCRIPTION="A toolkit for building high-level compound widgets in Python using the Tkinter module."
HOMEPAGE="http://pmw.sourceforge.net/"
SRC_URI="mirror://sourceforge/${PN}/${MY_P}.tar.gz"
LICENSE="BSD"
SLOT="0"
KEYWORDS="alpha amd64 ia64 ppc sparc x86 ~amd64-linux ~x86-linux"
IUSE="doc examples"
DEPEND="dev-lang/python-tk"
RDEPEND=""
S="${WORKDIR}/${MY_P}/src"
DOCS="Pmw/README"
PYTHON_MODNAME="Pmw"
src_prepare() {
distutils_src_prepare
epatch "${FILESDIR}/${P}-install-no-docs.patch"
epatch "${FILESDIR}/${PV}-python2.5.patch"
}
src_install() {
distutils_src_install
local DIR="${PYTHON_MODNAME}/Pmw_1_3"
if use doc; then
dohtml -a html,gif,py "${DIR}"/doc/* || die "Installation of documentation failed"
fi
if use examples; then
insinto "/usr/share/doc/${PF}/examples"
doins "${DIR}"/demos/* || die "Installation of demos failed"
fi
}
dev-python/urlgrabber/Manifest
-4
View File
@@ -1,4 +0,0 @@
AUX urlgrabber-HEAD.patch 28240 RMD160 faa21f43bc13f506c151dc698d18382e6bbd72d2 SHA1 04d29393e3ea27115410bb10a563815afb94f600 SHA256 0c177042052b51cfd69894f3f6c24d0d041affdaf1cdeb2baeade0d746b7c4ff
DIST urlgrabber-3.9.1.tar.gz 72343 RMD160 377196ddb01f6cd4903e063e8e9cba40a0f80fff SHA1 937c7ad3177539c337d4010e0175b2c7bd874938 SHA256 4437076c8708e5754ea04540e46c7f4f233734ee3590bb8a96389264fb0650d0
EBUILD urlgrabber-3.9.1.ebuild 722 RMD160 36c30126d4e9bbd3fbfe04f32c1812e954a88ff2 SHA1 99f52a4560cc78fd3960ad20301986a710daea63 SHA256 552e29b1a5f1c5508a57139788aa9b611b998bdb2ed6aa3ec5e8532725cb2464
MISC metadata.xml 1400 RMD160 6bed3463a85241b65f5384e6784477e63e924ee9 SHA1 f4d93729f1f20e8b93919047cf6f5c369e2b0ea3 SHA256 b66c06a0e5f9f2dc73e7c18d683d4bd13abbd8f5c44f1f60614fedc9f094e4c7
dev-python/urlgrabber/files/urlgrabber-HEAD.patch
-701
View File
@@ -1,701 +0,0 @@
diff --git a/scripts/urlgrabber b/scripts/urlgrabber
index 518e512..09cd896 100644
--- a/scripts/urlgrabber
+++ b/scripts/urlgrabber
@@ -115,6 +115,7 @@ options:
including quotes in the case of strings.
e.g. --user_agent='"foobar/2.0"'
+ --output FILE
-o FILE write output to FILE, otherwise the basename of the
url will be used
-O print the names of saved files to STDOUT
@@ -170,12 +171,17 @@ class client_options:
return ug_options, ug_defaults
def process_command_line(self):
- short_options = 'vd:hoOpD'
+ short_options = 'vd:ho:OpD'
long_options = ['profile', 'repeat=', 'verbose=',
- 'debug=', 'help', 'progress']
+ 'debug=', 'help', 'progress', 'output=']
ug_long = [ o + '=' for o in self.ug_options ]
- optlist, args = getopt.getopt(sys.argv[1:], short_options,
- long_options + ug_long)
+ try:
+ optlist, args = getopt.getopt(sys.argv[1:], short_options,
+ long_options + ug_long)
+ except getopt.GetoptError, e:
+ print >>sys.stderr, "Error:", e
+ self.help([], ret=1)
+
self.verbose = 0
self.debug = None
self.outputfile = None
@@ -193,6 +199,7 @@ class client_options:
if o == '--verbose': self.verbose = v
if o == '-v': self.verbose += 1
if o == '-o': self.outputfile = v
+ if o == '--output': self.outputfile = v
if o == '-p' or o == '--progress': self.progress = 1
if o == '-d' or o == '--debug': self.debug = v
if o == '--profile': self.profile = 1
@@ -222,7 +229,7 @@ class client_options:
print "ERROR: cannot use -o when grabbing multiple files"
sys.exit(1)
- def help(self, args):
+ def help(self, args, ret=0):
if not args:
print MAINHELP
else:
@@ -234,7 +241,7 @@ class client_options:
self.help_ug_option(a)
else:
print 'ERROR: no help on command "%s"' % a
- sys.exit(0)
+ sys.exit(ret)
def help_doc(self):
print __doc__
diff --git a/test/base_test_code.py b/test/base_test_code.py
index 50c6348..5fb43f9 100644
--- a/test/base_test_code.py
+++ b/test/base_test_code.py
@@ -1,6 +1,6 @@
from munittest import *
-base_http = 'http://www.linux.duke.edu/projects/urlgrabber/test/'
+base_http = 'http://urlgrabber.baseurl.org/test/'
base_ftp = 'ftp://localhost/test/'
# set to a proftp server only. we're working around a couple of
diff --git a/urlgrabber/byterange.py b/urlgrabber/byterange.py
index 3e5f3b7..8eeaeda 100644
--- a/urlgrabber/byterange.py
+++ b/urlgrabber/byterange.py
@@ -68,7 +68,7 @@ class HTTPRangeHandler(urllib2.BaseHandler):
def http_error_416(self, req, fp, code, msg, hdrs):
# HTTP's Range Not Satisfiable error
- raise RangeError('Requested Range Not Satisfiable')
+ raise RangeError(9, 'Requested Range Not Satisfiable')
class HTTPSRangeHandler(HTTPRangeHandler):
""" Range Header support for HTTPS. """
@@ -208,7 +208,7 @@ class RangeableFileObject:
bufsize = offset - pos
buf = self.fo.read(bufsize)
if len(buf) != bufsize:
- raise RangeError('Requested Range Not Satisfiable')
+ raise RangeError(9, 'Requested Range Not Satisfiable')
pos+= bufsize
class FileRangeHandler(urllib2.FileHandler):
@@ -238,7 +238,7 @@ class FileRangeHandler(urllib2.FileHandler):
(fb,lb) = brange
if lb == '': lb = size
if fb < 0 or fb > size or lb > size:
- raise RangeError('Requested Range Not Satisfiable')
+ raise RangeError(9, 'Requested Range Not Satisfiable')
size = (lb - fb)
fo = RangeableFileObject(fo, (fb,lb))
headers = mimetools.Message(StringIO(
@@ -318,12 +318,12 @@ class FTPRangeHandler(urllib2.FTPHandler):
(fb,lb) = range_tup
if lb == '':
if retrlen is None or retrlen == 0:
- raise RangeError('Requested Range Not Satisfiable due to unobtainable file length.')
+ raise RangeError(9, 'Requested Range Not Satisfiable due to unobtainable file length.')
lb = retrlen
retrlen = lb - fb
if retrlen < 0:
# beginning of range is larger than file
- raise RangeError('Requested Range Not Satisfiable')
+ raise RangeError(9, 'Requested Range Not Satisfiable')
else:
retrlen = lb - fb
fp = RangeableFileObject(fp, (0,retrlen))
@@ -458,6 +458,6 @@ def range_tuple_normalize(range_tup):
# check if range is over the entire file
if (fb,lb) == (0,''): return None
# check that the range is valid
- if lb < fb: raise RangeError('Invalid byte range: %s-%s' % (fb,lb))
+ if lb < fb: raise RangeError(9, 'Invalid byte range: %s-%s' % (fb,lb))
return (fb,lb)
diff --git a/urlgrabber/grabber.py b/urlgrabber/grabber.py
index e090e90..b2770c5 100644
--- a/urlgrabber/grabber.py
+++ b/urlgrabber/grabber.py
@@ -68,14 +68,14 @@ GENERAL ARGUMENTS (kwargs)
(which can be set on default_grabber.throttle) is used. See
BANDWIDTH THROTTLING for more information.
- timeout = None
+ timeout = 300
- a positive float expressing the number of seconds to wait for socket
- operations. If the value is None or 0.0, socket operations will block
- forever. Setting this option causes urlgrabber to call the settimeout
- method on the Socket object used for the request. See the Python
- documentation on settimeout for more information.
- http://www.python.org/doc/current/lib/socket-objects.html
+ a positive integer expressing the number of seconds to wait before
+ timing out attempts to connect to a server. If the value is None
+ or 0, connection attempts will not time out. The timeout is passed
+ to the underlying pycurl object as its CONNECTTIMEOUT option, see
+ the curl documentation on CURLOPT_CONNECTTIMEOUT for more information.
+ http://curl.haxx.se/libcurl/c/curl_easy_setopt.html#CURLOPTCONNECTTIMEOUT
bandwidth = 0
@@ -198,6 +198,12 @@ GENERAL ARGUMENTS (kwargs)
control, you should probably subclass URLParser and pass it in via
the 'urlparser' option.
+ username = None
+ username to use for simple http auth - is automatically quoted for special characters
+
+ password = None
+ password to use for simple http auth - is automatically quoted for special characters
+
ssl_ca_cert = None
this option can be used if M2Crypto is available and will be
@@ -248,6 +254,11 @@ GENERAL ARGUMENTS (kwargs)
Maximum size (in bytes) of the headers.
+ self.ip_resolve = 'whatever'
+
+ What type of name to IP resolving to use, default is to do both IPV4 and
+ IPV6.
+
RETRY RELATED ARGUMENTS
@@ -420,6 +431,7 @@ import time
import string
import urllib
import urllib2
+from httplib import responses
import mimetools
import thread
import types
@@ -439,6 +451,12 @@ try:
except:
__version__ = '???'
+try:
+ # this part isn't going to do much - need to talk to gettext
+ from i18n import _
+except ImportError, msg:
+ def _(st): return st
+
########################################################################
# functions for debugging output. These functions are here because they
# are also part of the module initialization.
@@ -527,6 +545,22 @@ def _(st):
# END MODULE INITIALIZATION
########################################################################
+########################################################################
+# UTILITY FUNCTIONS
+########################################################################
+
+# These functions are meant to be utilities for the urlgrabber library to use.
+
+def _to_utf8(obj, errors='replace'):
+ '''convert 'unicode' to an encoded utf-8 byte string '''
+ # stolen from yum.i18n
+ if isinstance(obj, unicode):
+ obj = obj.encode('utf-8', errors)
+ return obj
+
+########################################################################
+# END UTILITY FUNCTIONS
+########################################################################
class URLGrabError(IOError):
@@ -662,6 +696,7 @@ class URLParser:
opts.quote = 0 --> do not quote it
opts.quote = None --> guess
"""
+ url = _to_utf8(url)
quote = opts.quote
if opts.prefix:
@@ -800,6 +835,7 @@ class URLGrabberOptions:
self.close_connection = 0
self.range = None
self.user_agent = 'urlgrabber/%s' % __version__
+ self.ip_resolve = None
self.keepalive = 1
self.proxies = None
self.reget = None
@@ -808,13 +844,15 @@ class URLGrabberOptions:
self.prefix = None
self.opener = None
self.cache_openers = True
- self.timeout = None
+ self.timeout = 300
self.text = None
self.http_headers = None
self.ftp_headers = None
self.data = None
self.urlparser = URLParser()
self.quote = None
+ self.username = None
+ self.password = None
self.ssl_ca_cert = None # sets SSL_CAINFO - path to certdb
self.ssl_context = None # no-op in pycurl
self.ssl_verify_peer = True # check peer's cert for authenticityb
@@ -846,7 +884,7 @@ class URLGrabberOptions:
s = s + indent + '}'
return s
-class URLGrabber:
+class URLGrabber(object):
"""Provides easy opening of URLs with a variety of options.
All options are specified as kwargs. Options may be specified when
@@ -931,6 +969,9 @@ class URLGrabber:
(scheme, host, path, parm, query, frag) = parts
if filename is None:
filename = os.path.basename( urllib.unquote(path) )
+ if not filename:
+ # This is better than nothing.
+ filename = 'index.html'
if scheme == 'file' and not opts.copy_local:
# just return the name of the local file - don't make a
# copy currently
@@ -1030,7 +1071,7 @@ class URLGrabber:
default_grabber = URLGrabber()
-class PyCurlFileObject():
+class PyCurlFileObject(object):
def __init__(self, url, filename, opts):
self.fo = None
self._hdr_dump = ''
@@ -1052,9 +1093,15 @@ class PyCurlFileObject():
self._reget_length = 0
self._prog_running = False
self._error = (None, None)
- self.size = None
+ self.size = 0
+ self._hdr_ended = False
self._do_open()
+
+ def geturl(self):
+ """ Provide the geturl() method, used to be got from
+ urllib.addinfourl, via. urllib.URLopener.* """
+ return self.url
def __getattr__(self, name):
"""This effectively allows us to wrap at the instance level.
@@ -1085,9 +1132,14 @@ class PyCurlFileObject():
return -1
def _hdr_retrieve(self, buf):
+ if self._hdr_ended:
+ self._hdr_dump = ''
+ self.size = 0
+ self._hdr_ended = False
+
if self._over_max_size(cur=len(self._hdr_dump),
max_size=self.opts.max_header_size):
- return -1
+ return -1
try:
self._hdr_dump += buf
# we have to get the size before we do the progress obj start
@@ -1104,7 +1156,17 @@ class PyCurlFileObject():
s = parse150(buf)
if s:
self.size = int(s)
-
+
+ if buf.lower().find('location') != -1:
+ location = ':'.join(buf.split(':')[1:])
+ location = location.strip()
+ self.scheme = urlparse.urlsplit(location)[0]
+ self.url = location
+
+ if len(self._hdr_dump) != 0 and buf == '\r\n':
+ self._hdr_ended = True
+ if DEBUG: DEBUG.info('header ended:')
+
return len(buf)
except KeyboardInterrupt:
return pycurl.READFUNC_ABORT
@@ -1113,8 +1175,10 @@ class PyCurlFileObject():
if self._parsed_hdr:
return self._parsed_hdr
statusend = self._hdr_dump.find('\n')
+ statusend += 1 # ridiculous as it may seem.
hdrfp = StringIO()
hdrfp.write(self._hdr_dump[statusend:])
+ hdrfp.seek(0)
self._parsed_hdr = mimetools.Message(hdrfp)
return self._parsed_hdr
@@ -1136,11 +1200,21 @@ class PyCurlFileObject():
self.curl_obj.setopt(pycurl.PROGRESSFUNCTION, self._progress_update)
self.curl_obj.setopt(pycurl.FAILONERROR, True)
self.curl_obj.setopt(pycurl.OPT_FILETIME, True)
+ self.curl_obj.setopt(pycurl.FOLLOWLOCATION, True)
if DEBUG:
self.curl_obj.setopt(pycurl.VERBOSE, True)
if opts.user_agent:
self.curl_obj.setopt(pycurl.USERAGENT, opts.user_agent)
+ if opts.ip_resolve:
+ # Default is: IPRESOLVE_WHATEVER
+ ipr = opts.ip_resolve.lower()
+ if ipr == 'whatever': # Do we need this?
+ self.curl_obj.setopt(pycurl.IPRESOLVE,pycurl.IPRESOLVE_WHATEVER)
+ if ipr == 'ipv4':
+ self.curl_obj.setopt(pycurl.IPRESOLVE, pycurl.IPRESOLVE_V4)
+ if ipr == 'ipv6':
+ self.curl_obj.setopt(pycurl.IPRESOLVE, pycurl.IPRESOLVE_V6)
# maybe to be options later
self.curl_obj.setopt(pycurl.FOLLOWLOCATION, True)
@@ -1148,9 +1222,11 @@ class PyCurlFileObject():
# timeouts
timeout = 300
- if opts.timeout:
- timeout = int(opts.timeout)
- self.curl_obj.setopt(pycurl.CONNECTTIMEOUT, timeout)
+ if hasattr(opts, 'timeout'):
+ timeout = int(opts.timeout or 0)
+ self.curl_obj.setopt(pycurl.CONNECTTIMEOUT, timeout)
+ self.curl_obj.setopt(pycurl.LOW_SPEED_LIMIT, 1)
+ self.curl_obj.setopt(pycurl.LOW_SPEED_TIME, timeout)
# ssl options
if self.scheme == 'https':
@@ -1203,12 +1279,19 @@ class PyCurlFileObject():
if proxy == '_none_': proxy = ""
self.curl_obj.setopt(pycurl.PROXY, proxy)
- # FIXME username/password/auth settings
+ if opts.username and opts.password:
+ if self.scheme in ('http', 'https'):
+ self.curl_obj.setopt(pycurl.HTTPAUTH, pycurl.HTTPAUTH_ANY)
+
+ if opts.username and opts.password:
+ # apparently when applying them as curlopts they do not require quoting of any kind
+ userpwd = '%s:%s' % (opts.username, opts.password)
+ self.curl_obj.setopt(pycurl.USERPWD, userpwd)
#posts - simple - expects the fields as they are
if opts.data:
self.curl_obj.setopt(pycurl.POST, True)
- self.curl_obj.setopt(pycurl.POSTFIELDS, self._to_utf8(opts.data))
+ self.curl_obj.setopt(pycurl.POSTFIELDS, _to_utf8(opts.data))
# our url
self.curl_obj.setopt(pycurl.URL, self.url)
@@ -1228,12 +1311,14 @@ class PyCurlFileObject():
code = self.http_code
errcode = e.args[0]
+ errurl = urllib.unquote(self.url)
+
if self._error[0]:
errcode = self._error[0]
if errcode == 23 and code >= 200 and code < 299:
- err = URLGrabError(15, _('User (or something) called abort %s: %s') % (self.url, e))
- err.url = self.url
+ err = URLGrabError(15, _('User (or something) called abort %s: %s') % (errurl, e))
+ err.url = errurl
# this is probably wrong but ultimately this is what happens
# we have a legit http code and a pycurl 'writer failed' code
@@ -1244,23 +1329,23 @@ class PyCurlFileObject():
raise KeyboardInterrupt
elif errcode == 28:
- err = URLGrabError(12, _('Timeout on %s: %s') % (self.url, e))
- err.url = self.url
+ err = URLGrabError(12, _('Timeout on %s: %s') % (errurl, e))
+ err.url = errurl
raise err
elif errcode == 35:
msg = _("problem making ssl connection")
err = URLGrabError(14, msg)
- err.url = self.url
+ err.url = errurl
raise err
elif errcode == 37:
- msg = _("Could not open/read %s") % (self.url)
+ msg = _("Could not open/read %s") % (errurl)
err = URLGrabError(14, msg)
- err.url = self.url
+ err.url = errurl
raise err
elif errcode == 42:
- err = URLGrabError(15, _('User (or something) called abort %s: %s') % (self.url, e))
- err.url = self.url
+ err = URLGrabError(15, _('User (or something) called abort %s: %s') % (errurl, e))
+ err.url = errurl
# this is probably wrong but ultimately this is what happens
# we have a legit http code and a pycurl 'writer failed' code
# which almost always means something aborted it from outside
@@ -1272,33 +1357,93 @@ class PyCurlFileObject():
elif errcode == 58:
msg = _("problem with the local client certificate")
err = URLGrabError(14, msg)
- err.url = self.url
+ err.url = errurl
raise err
elif errcode == 60:
- msg = _("client cert cannot be verified or client cert incorrect")
+ msg = _("Peer cert cannot be verified or peer cert invalid")
err = URLGrabError(14, msg)
- err.url = self.url
+ err.url = errurl
raise err
elif errcode == 63:
if self._error[1]:
msg = self._error[1]
else:
- msg = _("Max download size exceeded on %s") % (self.url)
+ msg = _("Max download size exceeded on %s") % ()
err = URLGrabError(14, msg)
- err.url = self.url
+ err.url = errurl
raise err
elif str(e.args[1]) == '' and self.http_code != 0: # fake it until you make it
- msg = 'HTTP Error %s : %s ' % (self.http_code, self.url)
+ if self.scheme in ['http', 'https']:
+ if self.http_code in responses:
+ resp = responses[self.http_code]
+ msg = 'HTTP Error %s - %s : %s' % (self.http_code, resp, errurl)
+ else:
+ msg = 'HTTP Error %s : %s ' % (self.http_code, errurl)
+ elif self.scheme in ['ftp']:
+ msg = 'FTP Error %s : %s ' % (self.http_code, errurl)
+ else:
+ msg = "Unknown Error: URL=%s , scheme=%s" % (errurl, self.scheme)
else:
- msg = 'PYCURL ERROR %s - "%s"' % (errcode, str(e.args[1]))
+ pyerr2str = { 5 : _("Couldn't resolve proxy"),
+ 6 : _("Couldn't resolve host"),
+ 7 : _("Couldn't connect"),
+ 8 : _("Bad reply to FTP server"),
+ 9 : _("Access denied"),
+ 11 : _("Bad reply to FTP pass"),
+ 13 : _("Bad reply to FTP pasv"),
+ 14 : _("Bad reply to FTP 227"),
+ 15 : _("Couldn't get FTP host"),
+ 17 : _("Couldn't set FTP type"),
+ 18 : _("Partial file"),
+ 19 : _("FTP RETR command failed"),
+ 22 : _("HTTP returned error"),
+ 23 : _("Write error"),
+ 25 : _("Upload failed"),
+ 26 : _("Read error"),
+ 27 : _("Out of Memory"),
+ 28 : _("Operation timed out"),
+ 30 : _("FTP PORT command failed"),
+ 31 : _("FTP REST command failed"),
+ 33 : _("Range failed"),
+ 34 : _("HTTP POST failed"),
+ 35 : _("SSL CONNECT failed"),
+ 36 : _("Couldn't resume download"),
+ 37 : _("Couldn't read file"),
+ 42 : _("Aborted by callback"),
+ 47 : _("Too many redirects"),
+ 51 : _("Peer certificate failed verification"),
+ 53 : _("SSL engine not found"),
+ 54 : _("SSL engine set failed"),
+ 55 : _("Network error send()"),
+ 56 : _("Network error recv()"),
+ 58 : _("Local certificate failed"),
+ 59 : _("SSL set cipher failed"),
+ 60 : _("Local CA certificate failed"),
+ 61 : _("HTTP bad transfer encoding"),
+ 63 : _("Maximum file size exceeded"),
+ 64 : _("FTP SSL failed"),
+ 67 : _("Authentication failure"),
+ 70 : _("Out of disk space on server"),
+ 73 : _("Remove file exists"),
+ }
+ errstr = str(e.args[1])
+ if not errstr:
+ errstr = pyerr2str.get(errcode, '<Unknown>')
+ msg = 'curl#%s - "%s"' % (errcode, errstr)
code = errcode
err = URLGrabError(14, msg)
err.code = code
err.exception = e
raise err
+ else:
+ if self._error[1]:
+ msg = self._error[1]
+ err = URLGRabError(14, msg)
+ err.url = urllib.unquote(self.url)
+ raise err
def _do_open(self):
self.curl_obj = _curl_cache
@@ -1333,7 +1478,11 @@ class PyCurlFileObject():
if self.opts.range:
rt = self.opts.range
- if rt[0]: rt = (rt[0] + reget_length, rt[1])
+
+ if rt[0] is None:
+ rt = (0, rt[1])
+ rt = (rt[0] + reget_length, rt[1])
+
if rt:
header = range_tuple_to_header(rt)
@@ -1434,9 +1583,13 @@ class PyCurlFileObject():
#fh, self._temp_name = mkstemp()
#self.fo = open(self._temp_name, 'wb')
-
- self._do_perform()
-
+ try:
+ self._do_perform()
+ except URLGrabError, e:
+ self.fo.flush()
+ self.fo.close()
+ raise e
+
if _was_filename:
@@ -1446,9 +1599,23 @@ class PyCurlFileObject():
# set the time
mod_time = self.curl_obj.getinfo(pycurl.INFO_FILETIME)
if mod_time != -1:
- os.utime(self.filename, (mod_time, mod_time))
+ try:
+ os.utime(self.filename, (mod_time, mod_time))
+ except OSError, e:
+ err = URLGrabError(16, _(\
+ 'error setting timestamp on file %s from %s, OSError: %s')
+ % (self.filename, self.url, e))
+ err.url = self.url
+ raise err
# re open it
- self.fo = open(self.filename, 'r')
+ try:
+ self.fo = open(self.filename, 'r')
+ except IOError, e:
+ err = URLGrabError(16, _(\
+ 'error opening file from %s, IOError: %s') % (self.url, e))
+ err.url = self.url
+ raise err
+
else:
#self.fo = open(self._temp_name, 'r')
self.fo.seek(0)
@@ -1532,11 +1699,14 @@ class PyCurlFileObject():
def _over_max_size(self, cur, max_size=None):
if not max_size:
- max_size = self.size
- if self.opts.size: # if we set an opts size use that, no matter what
- max_size = self.opts.size
+ if not self.opts.size:
+ max_size = self.size
+ else:
+ max_size = self.opts.size
+
if not max_size: return False # if we have None for all of the Max then this is dumb
- if cur > max_size + max_size*.10:
+
+ if cur > int(float(max_size) * 1.10):
msg = _("Downloaded more than max size for %s: %s > %s") \
% (self.url, cur, max_size)
@@ -1544,13 +1714,6 @@ class PyCurlFileObject():
return True
return False
- def _to_utf8(self, obj, errors='replace'):
- '''convert 'unicode' to an encoded utf-8 byte string '''
- # stolen from yum.i18n
- if isinstance(obj, unicode):
- obj = obj.encode('utf-8', errors)
- return obj
-
def read(self, amt=None):
self._fill_buffer(amt)
if amt is None:
@@ -1582,9 +1745,21 @@ class PyCurlFileObject():
self.opts.progress_obj.end(self._amount_read)
self.fo.close()
-
+ def geturl(self):
+ """ Provide the geturl() method, used to be got from
+ urllib.addinfourl, via. urllib.URLopener.* """
+ return self.url
+
_curl_cache = pycurl.Curl() # make one and reuse it over and over and over
+def reset_curl_obj():
+ """To make sure curl has reread the network/dns info we force a reload"""
+ global _curl_cache
+ _curl_cache.close()
+ _curl_cache = pycurl.Curl()
+
+
+
#####################################################################
# DEPRECATED FUNCTIONS
diff --git a/urlgrabber/mirror.py b/urlgrabber/mirror.py
index dad410b..8731aed 100644
--- a/urlgrabber/mirror.py
+++ b/urlgrabber/mirror.py
@@ -90,7 +90,7 @@ CUSTOMIZATION
import random
import thread # needed for locking to make this threadsafe
-from grabber import URLGrabError, CallbackObject, DEBUG
+from grabber import URLGrabError, CallbackObject, DEBUG, _to_utf8
def _(st):
return st
@@ -263,7 +263,8 @@ class MirrorGroup:
def _parse_mirrors(self, mirrors):
parsed_mirrors = []
for m in mirrors:
- if type(m) == type(''): m = {'mirror': m}
+ if isinstance(m, basestring):
+ m = {'mirror': _to_utf8(m)}
parsed_mirrors.append(m)
return parsed_mirrors
diff --git a/urlgrabber/progress.py b/urlgrabber/progress.py
index dd07c6a..45eb248 100644
--- a/urlgrabber/progress.py
+++ b/urlgrabber/progress.py
@@ -658,6 +658,8 @@ def format_time(seconds, use_hours=0):
if seconds is None or seconds < 0:
if use_hours: return '--:--:--'
else: return '--:--'
+ elif seconds == float('inf'):
+ return 'Infinite'
else:
seconds = int(seconds)
minutes = seconds / 60
dev-python/urlgrabber/metadata.xml
-33
View File
@@ -1,33 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE pkgmetadata SYSTEM "http://www.gentoo.org/dtd/metadata.dtd">
<pkgmetadata>
<herd>python</herd>
<longdescription>
A high-level cross-protocol url-grabber.
Using urlgrabber, data can be fetched in three basic ways:
urlgrab(url) copy the file to the local filesystem
urlopen(url) open the remote file and return a file object (like urllib2.urlopen)
urlread(url) return the contents of the file as a string
When using these functions (or methods), urlgrabber supports the following
features:
* identical behavior for http://, ftp://, and file:// urls
* http keepalive - faster downloads of many files by using only a single
connection
* byte ranges - fetch only a portion of the file
* reget - for a urlgrab, resume a partial download
* progress meters - the ability to report download progress automatically,
even when using urlopen!
* throttling - restrict bandwidth usage
* retries - automatically retry a download if it fails. The number of retries
and failure types are configurable.
* authenticated server access for http and ftp
* proxy support - support for authenticated http and ftp proxies
* mirror groups - treat a list of mirrors as a single source, automatically
switching mirrors if there is a failure.
</longdescription>
</pkgmetadata>
dev-python/urlgrabber/urlgrabber-3.9.1.ebuild
-28
View File
@@ -1,28 +0,0 @@
# Copyright 1999-2010 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: /var/cvsroot/gentoo-x86/dev-python/urlgrabber/urlgrabber-3.9.1.ebuild,v 1.8 2010/12/27 21:05:25 ranger Exp $
EAPI="3"
PYTHON_DEPEND="2:2.5"
SUPPORT_PYTHON_ABIS="1"
RESTRICT_PYTHON_ABIS="2.4 3.*"
inherit eutils distutils
DESCRIPTION="Python module for downloading files"
HOMEPAGE="http://urlgrabber.baseurl.org"
SRC_URI="http://urlgrabber.baseurl.org/download/${P}.tar.gz"
LICENSE="LGPL-2.1"
SLOT="0"
KEYWORDS="amd64 ~ia64 ppc ppc64 x86 ~x86-fbsd"
IUSE=""
DEPEND="dev-python/pycurl"
RDEPEND="${DEPEND}"
RESTRICT="test"
src_prepare() {
distutils_src_prepare
epatch "${FILESDIR}/${PN}-HEAD.patch"
}