-=[ Mr. Bumblebee ]=-
_Indonesia_
:Nc @@ so d Z d d l m Z d d d d d g Z d d l m Z d d l m Z d d
l Z d d
l Z d d l
m Z e e d d d
l
m Z m Z d d l m Z d d f Z d Z d Z d Z d Z e j d Z e j d Z d Z d Z d e f d YZ d e f d YZ d e f d YZ d e f d YZ d e f d YZ! d
S( s Indexing facilities.i ( t absolute_importt CombinedGraphIndext
GraphIndext GraphIndexBuildert GraphIndexPrefixAdaptert InMemoryGraphIndex( t bisect_right( t StringION( t lazy_importsX
from bzrlib import (
bisect_multi,
revision as _mod_revision,
trace,
)
( t debugt errors( t StaticTuplei s
key_elements=s len=s node_ref_lists=s Bazaar Graph Index 1
s [
]s [
]c C@ s | | j | g k S( s Check if this index has one key.
If it's possible to check for multiple keys at once through
calling get_parent_map that should be faster.
( t get_parent_map( t selft key( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyt _has_key_from_parent_map; s c C@ s t | t | j | S( N( t setR ( R
t keys( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyt _missing_keys_from_parent_mapD s c B@ s e Z d Z d d d Z d Z d Z d Z d Z d Z d d Z d
Z
d Z d d d Z
d
Z RS( s A builder that can build a GraphIndex.
The resulting graph has the structure::
_SIGNATURE OPTIONS NODES NEWLINE
_SIGNATURE := 'Bazaar Graph Index 1' NEWLINE
OPTIONS := 'node_ref_lists=' DIGITS NEWLINE
NODES := NODE*
NODE := KEY NULL ABSENT? NULL REFERENCES NULL VALUE NEWLINE
KEY := Not-whitespace-utf8
ABSENT := 'a'
REFERENCES := REFERENCE_LIST (TAB REFERENCE_LIST){node_ref_lists - 1}
REFERENCE_LIST := (REFERENCE (CR REFERENCE)*)?
REFERENCE := DIGITS ; digits is the byte offset in the index of the
; referenced key.
VALUE := no-newline-no-null-bytes
i i c C@ sF | | _ i | _ t | _ d | _ | | _ t | _ t | _
d S( s Create a GraphIndex builder.
:param reference_lists: The number of node references lists for each
entry.
:param key_elements: The number of bytestrings in each key.
N( t reference_listst _nodesR t _absent_keyst Nonet
_nodes_by_keyt _key_lengtht Falset _optimize_for_sizet Truet _combine_backing_indices( R
R t key_elements( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyt __init__[ s c C@ s t | t t f k r* t j | n | j t | k rQ t j | n x< | D]4 } | sz t j | d k rX t j | qX qX Wd S( s; Raise BadIndexKey if key is not a valid key for this index.N(
t typet tupleR R
t BadIndexKeyR t lent _whitespace_ret searchR ( R
R t element( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyt
_check_keyl s
c C@ sq t } t } | j d k rf x: | j D], } | j | d | j | d d q. W| | St Sd S( s> Return references that are not present in this index.
i i N( R R t iter_all_entriest addt update( R
R t refst node( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyt _external_referencesv s c C@ s# | j d k r i } | j r x | j j D]e \ } \ } } } | rO q. n | } x$ | d D] } | j | i } q` W| | | f | | d <q. Wnv xs | j j D]b \ } \ } } } | r q n | } x$ | d D] } | j | i } q W| | f | | d <q W| | _ n | j S( Ni( R R R R t iteritemst
setdefault( R
t nodes_by_keyR t absentt
referencest valuet key_dictt subkey( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyt _get_nodes_by_key s& % % c C@ s | j d k r d S| j } | j r: t | | | } n t | | } x$ | d D] } | j | i } qT W| | | d <d S( s Update the _nodes_by_key dict with a new key.
For a key of (foo, bar, baz) create
_nodes_by_key[foo][bar][baz] = key_value
Ni( R R R R R. ( R
R R2 t node_refsR3 t key_valueR4 ( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyt _update_nodes_by_key s c
C@ s t j } | j | t j | d k r= t j | n t | | j k rd t j | n g } g } x | D]{ } x: | D]2 } | | j
k r | j | | j | q q W| g | D] } | | j ^ q } | j | qw W| | | f S( sA Check that 'key' and 'references' are all valid.
:param key: A key tuple. Must conform to the key interface (be a tuple,
be of the right length, not have any whitespace or nulls in any key
element.)
:param references: An iterable of reference lists. Something like
[[(ref, key)], [(ref, key), (other, key)]]
:param value: The value associate with this key. Must not contain
newlines or null characters.
:return: (node_refs, absent_references)
* node_refs: basically a packed form of 'references' where all
iterables are tuples
* absent_references: reference keys that are not in self._nodes.
This may contain duplicates if the same key is referenced in
multiple lists.
N(
R t
from_sequenceR& t _newline_null_reR$ R R
t
BadIndexValueR" R R t appendt intern(
R
R R1 R2 t as_stR6 t absent_referencest reference_listt referencet ref( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyt _check_key_ref_value s"
%c C@ s | j | | | \ } } | | j k rV | j | d d k rV t j | | n x$ | D] } d d d f | j | <q] W| j j | | j j | d | | f | j | <| j d k r | j d k r | j
| | | n d S( s Add a node to the index.
:param key: The key. keys are non-empty tuples containing
as many whitespace-free utf8 bytestrings as the key length
defined for this index.
:param references: An iterable of iterables of keys. Each is a
reference to another key.
:param value: The value to associate with the key. It may be any
bytes as long as it does not contain \0 or \n.
i t at i N( ( RC R R
t BadIndexDuplicateKeyR R) t discardR R R R8 ( R
R R2 R1 R6 R? RA ( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyt add_node s &
c C@ s d S( s See GraphIndex.clear_cache()
This is a no-op, but we need the api to conform to a generic 'Index'
abstraction.
N( ( R
( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyt clear_cache s c C@ sg t g } | j t t | j d | j t t | j d t | j t | j } | j t
t | d t d | D } t | j j
} d } | j rMg } | } d } x | D] \ } \ }
} } | j | | | f | t d | D 7} | j d k r7| | j d 7} n | t | d d 7} |
rb| d 7} q | j r | | j d 7} x> | D]3 }
| t |
7} |
r| t |
d 7} qqWq q Wd } | | | } x, d | | k r| d 7} | | | } qW| d } i } x) | D]! \ } } } | | | | | <qWd | } n x | D] \ } \ }
} } g } xP | D]H }
g } x# |
D] } | j | | | qW| j d j | qvWd
j | } | j d | |
d j | | f qTW| j d t d
j | } | rct | j | k rct j d t | j | f n | S( s Finish the index.
:returns: cStringIO holding the full context of the index as it
should be written to disk.
s
c s@ s | ] } t | Vq d S( N( R" ( t .0t x( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pys <genexpr> s i c s@ s | ] } t | Vq d S( N( R" ( RJ R% ( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pys <genexpr>$ s i i i
s %%0%sds
t s %s %s %s %s
s RE sZ Failed index creation. Internal error: mismatched output length and expected length: %d %dN( t
_SIGNATURER< t _OPTION_NODE_REFSt strR t _OPTION_KEY_ELEMENTSR R" R R t _OPTION_LENt sumt sortedt itemsR t joinR t getvalueR
t BzrError( R
t linest key_countt
prefix_lengtht nodest expected_bytest key_offset_infot
non_ref_bytest total_referencesR R0 R1 R2 t ref_listt digitst possible_total_bytest
key_addressest
format_stringt flattened_referencest
ref_addressesRA t
string_keyt result( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyt finish sh
"
c C@ s4 | d k r | | _ n | d k r0 | | _ n d S( s Change how the builder tries to optimize the result.
:param for_size: Tell the builder to try and make the index as small as
possible.
:param combine_backing_indices: If the builder spills to disk to save
memory, should the on-disk indices be combined. Set to True if you
are going to be probing the index, but to False if you are not. (If
you are not querying, then the time spent combining is wasted.)
:return: None
N( R R R ( R
t for_sizet combine_backing_indices( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyt set_optimizeU s
c
C@ s t | } i } t } x | r t } xy | j | D]h \ } } } }
|
| } | | | <| j g | D] } | | k rp | ^ qp | j | j | q= W| } q W| | f S( s& See CombinedGraphIndex.find_ancestry()( R t iter_entriesR) t
difference(
R
R t ref_list_numt pendingt
parent_mapt missing_keyst next_pendingt _R R2 t ref_listst parent_keyst p( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyt
find_ancestryg s "
( N( t __name__t
__module__t __doc__R R& R, R5 R8 RC RH RI Ri R Rl Rx ( ( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyR H s
& \c B@ s e Z d Z e d d Z d Z d Z d Z d d Z d Z
d Z d Z d
Z
d Z d Z d
Z d Z d Z d Z d Z d Z d Z d Z d Z d Z d Z d Z d Z d Z d Z d Z d Z d Z! RS( s> An index for data with embedded graphs.
The index maps keys to a list of key reference lists, and a value.
Each node has the same number of key reference lists. Each key reference
list can be empty or an arbitrary length. The value is an opaque NULL
terminated string without any newlines. The storage of the index is
hidden in the interface: keys and key references are always tuples of
bytestrings, never the internal representation (e.g. dictionary offsets).
It is presumed that the index will not be mutated - it is static data.
Successive iter_all_entries calls will read the entire index each time.
Additionally, iter_entries calls will read the index linearly until the
desired keys are found. XXX: This must be fixed before the index is
suitable for production use. :XXX
i c C@ sp | | _ | | _ d | _ d | _ g | _ g | _ d | _ d | _ d | _ | | _
d | _ | | _ d S( s Open an index called name on transport.
:param transport: A bzrlib.transport.Transport.
:param name: A path to provide to transport API calls.
:param size: The size of the index in bytes. This is used for bisection
logic to perform partial index reads. While the size could be
obtained by statting the file this introduced an additional round
trip as well as requiring stat'able transports, both of which are
avoided by having it supplied. If size is None, then bisection
support will be disabled and accessing the index will just stream
all the data.
:param offset: Instead of starting the index data at offset 0, start it
at an arbitrary offset.
i N(
t
_transportt _nameR t
_bisect_nodesR t _parsed_byte_mapt _parsed_key_mapt
_key_countt _keys_by_offsetR t _sizet _bytes_readt _base_offset( R
t transportt namet sizet unlimited_cachet offset( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyR s c C@ sL t | t | k oK | j | j k oK | j | j k oK | j | j k S( s@ Equal when self and other were created with the same parameters.( R R| R} R ( R
t other( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyt __eq__ s c C@ s | j | S( N( R ( R
R ( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyt __ne__ s c C@ s# d | j j | j j | j f S( Ns %s(%r)( t __class__Ry R| t abspathR} ( R
( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyt __repr__ s c C@ s | j d k r d Sd t j k rD t j d | j j | j n | d k r | j j | j } | j
d k r t | j | j
} q n | j
| d | j | _ d } i | _ i | _ d | _ d } | j } | j j d } | j | d =| j | | \ } } } } xc | j j D]R \ } } } }
| rUq7n | j rv|
| j | f } n |
} | | j | <q7W| d k rt j | n d S( sY Buffer all the index data.
Mutates self._nodes and self.keys_by_offset.
Nt indexs Reading entire index %si i s
ii ( R R R t debug_flagst tracet mutterR| R R} t getR R t readt _read_prefixR t _expected_elementsR R t tellt splitt closet _parse_linest
itervaluest node_ref_listst _resolve_referencesR
t BadIndexData( R
t streamt
line_countt trailerst posRX Rt R R0 R1 R2 t
node_value( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyt _buffer_all s<
" c C@ s d S( s Clear out any cached/memoized values.
This can be called at any time, but generally it is used when we have
extracted some information, but don't expect to be requesting any more
from this index.
N( ( R
( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyRI s c C@ s | j | d | j k r9 t d | | j f n t } | j } xV | j D]H \ } \ } } | | } | j g | D] } | | k r | ^ q qX W| S( s> Return references that are not present in this index.
i s& No ref list %d, index has %d ref lists( R R t
ValueErrorR R R- R) ( R
Ro R* R[ R R2 Ru R` RB ( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyt external_references s
0c C@ s | j d k r i } | j r x | j j D]V \ } \ } } | } x$ | d D] } | j | i } qQ W| | | f | | d <q. Wna x^ | j j D]M \ } } | } x$ | d D] } | j | i } q W| | f | | d <q W| | _ n | j S( Ni( R R R R R- R. ( R
R/ R R2 R1 R3 R4 ( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyR5 s " c c@ s d t j k r" t j d d n | j d k r> | j n | j r xe | j j D]# \ } \ } } | | | | f VqW Wn. x+ | j j D] \ } } | | | f Vq Wd S( s Iterate over all keys within the index.
:return: An iterable of (index, key, value) or (index, key, value, reference_lists).
The former tuple is used when there are no reference lists in the
index, making the API compatible with simple key:value index types.
There is no defined order for the result iteration - it will be in
the most efficient order for the index.
t evili s- iter_all_entries scales with size of history.N( R R R t mutter_callsiteR R R R R- ( R
R R2 R ( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyR' s
" c C@ s | j t | j } | | j k sE t j | j t n | j } | j t sr t j
| n y t | t t d ! | _ Wn t
k
r t j
| n X| j } | j t s t j
| n y t | t t d ! | _ Wn t
k
r$t j
| n X| j } | j t sRt j
| n y t | t t d ! | _ Wn t
k
rt j
| n Xd S( Ni( R R" t
_signatureR
t BadIndexFormatSignatureR} R t readlinet
startswithRN t BadIndexOptionst intR R RP R RQ R ( R
R t signaturet options_line( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyR * s0
c C@ sR g } x? | D]7 } | j t g | D] } | j | d ^ q# q
Wt | S( sq Return the resolved key references for references.
References are resolved by looking up the location of the key in the
_keys_by_offset map and substituting the key name, preserving ordering.
:param references: An iterable of iterables of key locations. e.g.
[[123, 456], [123]]
:return: A tuple of tuples of keys.
i ( R< R R ( R
R1 R6 R` RB ( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyR D s
5c C@ sT t | | d } | d t | k rP | | d d | d k rP | d Sn | S( s Helper for the _parsed_*_index calls.
Given a range map - [(start, end), ...], finds the index of the range
in the map for key if it is in the map, and if it is not there, the
immediately preceeding range in the map.
i i ( R R" ( R
t range_mapR Rh ( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyt _find_indexS s
c C@ s | d f } | j | j | S( sg Return the index of the entry immediately before offset.
e.g. if the parsed map has regions 0,10 and 11,12 parsed, meaning that
there is one unparsed byte (the 11th, addressed as[10]). then:
asking for 0 will return 0
asking for 10 will return 0
asking for 11 will return 1
asking for 12 will return 1
i ( R R ( R
R R ( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyt _parsed_byte_indexa s
c C@ s | d f } | j | j | S( s Return the index of the entry immediately before key.
e.g. if the parsed map has regions (None, 'a') and ('b','c') parsed,
meaning that keys from None to 'a' inclusive, and 'b' to 'c' inclusive
have been parsed, then:
asking for '' will return 0
asking for 'a' will return 0
asking for 'b' will return 1
asking for 'e' will return 1
N( R R R ( R
R t
search_key( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyt _parsed_key_indexn s c C@ sf | j | } | t | j k r= | | j | d d k S| j | \ } } | | k oe | | k S( s' Returns True if offset has been parsed.i ( R R" R ( R
R R t startt end( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyt
_is_parsed| s
c c@ s | j } g | D] } | | k r | ^ q } | j rl xU | D]' } | | \ } } | | | | f Vq> Wn# x | D] } | | | | f Vqs Wd S( s2 Iterate over keys when the entire index is parsed.N( R R ( R
R R[ R R2 R6 ( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyt _iter_entries_from_total_buffer s %
c C@ s t | } | s g S| j d k rA | j d k rA | j n | j d k ry t | d | j k ry | j n | j d k r | j | Sd t j | j
| j | D Sd S( sH Iterate over keys within the index.
:param keys: An iterable providing the keys to be retrieved.
:return: An iterable as per iter_all_entries, but restricted to the
keys supplied. No additional keys will be returned, and every
key supplied that is in the index will be returned.
i c s@ s | ] } | d Vq d S( i N( ( RJ Rh ( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pys <genexpr> s N( R R R R R R" RY R t bisect_multit bisect_multi_bytest _lookup_keys_via_location( R
R ( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyRm s
+
c
c@ s\ t | } | s d S| j d k r2 | j n | j d k r x | D] } | d d k rp t j | n t | | j k r t j | n | j r | j | \ } } | | | | f VqH | | | j | f VqH Wd S| j } xe| D]]} | d d k r t j | n t | | j k rFt j | n | } t
| } yB x; t | r| d d k r| | d } | j d q^WWn t k
rq n Xt | rH| g } x | rD| j d } | j
j \ } } t | t k r | j | j qx | j D] } | f | Vq+WqWq | f | Vq Wd S( s Iterate over keys within the index using prefix matching.
Prefix matching is applied within the tuple of a key, not to within
the bytestring of each key element. e.g. if you have the keys ('foo',
'bar'), ('foobar', 'gam') and do a prefix search for ('foo', None) then
only the former key is returned.
WARNING: Note that this method currently causes a full index parse
unconditionally (which is reasonably appropriate as it is a means for
thunking many small indices into one larger one and still supplies
iter_all_entries at the thunk layer).
:param keys: An iterable providing the key prefixes to be retrieved.
Each key prefix takes the form of a tuple the length of a key, but
with the last N elements 'None' rather than a regular bytestring.
The first element cannot be 'None'.
:return: An iterable as per iter_all_entries, but restricted to the
keys with a matching prefix to those supplied. No additional keys
will be returned, and every match that is in the index will be
returned.
Ni i i( R R R R R R
R! R" R R5 t listt popt KeyErrorR- t nextR t dictt extendR (
R
R R R2 R6 R/ R3 t elementst dictst item( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyt iter_entries_prefix sP
c C@ s t } t } xQ | j | D]@ \ } } } }
|
| } | j | | | | <| j | q" W| j t | j | | j | } | S( s See BTreeIndex._find_ancestors.( R Rm R( R) Rn ( R
R Ro Rq Rr t
found_keyst search_keysR R R2 R* Rv ( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyt _find_ancestors s "
c C@ s) | j d k r" | j t g n | j S( so Return an estimate of the number of keys in this index.
For GraphIndex the estimate is exact.
N( R R t _read_and_parset
_HEADER_READV( R
( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyRY s c C@ sN g } x;| D]3\ } } | j r7 | | j k r7 q
n | j | } t | j r | j | d | k r | j | d | k s
| j | d | j k r q
n | j | } t | j r | j | d | k r | j | d | k r q
n d } | | | j k r | j | } n | d k r
| j | | f q
q
W| j d k rc| j t n | j
| g } | j d k r8x | D] \ } } | | j k r| j | | f t f q| j
r| j | \ } } | j | | f | | | | f f q| j | | f | | | j | f f qW| Sg }
t } x| D]\ } } | | j k rX| j
r)| j | \ } } g } x> | D]6 }
x- |
D]% } | | j k r| j | qqWqW| r| j | |
j | | f qNn | j | | f | | | | j | f f qN| j | | f | | | j | f f qNnv | j | } | j | d | k r| j | d | k s| j | d | j k r| j | | f t f qNn | j | } | | j | d k rd } n d
} | j | | f | f qNWg } xY | D]Q } d } | | | j k r]| j | } n | d k r.| j | | f q.q.W| j
| | j d k rxL |
D]D \ } } | j | \ } } | j | | f | | | | f f qW| SxU |
D]M \ } } | j | \ } } | j | | f | | | | j | f f qW| S( s Public interface for implementing bisection.
If _buffer_all has been called, then all the data for the index is in
memory, and this method should not be called, as it uses a separate
cache because it cannot pre-resolve all indices, which buffer_all does
for performance.
:param location_keys: A list of location(byte offset), key tuples.
:return: A list of (location_key, result) tuples as expected by
bzrlib.bisect_multi.bisect_multi_bytes.
i i i iN( R~ R R" R R R R R< R R R R R R R R R) R ( R
t
location_keyst readv_rangest locationR R t lengthRh R2 R* t pending_referencest pending_locationst wanted_locationsR` RB t direction( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyR s
) c C@ s | d t | j !} | | j k sC t j | j t n | t | j j } | d } | j t s t j | n y t
| t t | _ Wn t k
r t j | n X| d } | j t
s t j | n y t
| t t
| _ Wn t k
r4t j | n X| d } | j t s`t j | n y t
| t t | _ Wn t k
rt j | n Xt | t | d t | d t | d d } | j d d | d d | j | _ i | _ i | _ | | | f S( s Parse the header from a region of bytes.
:param bytes: The data to parse.
:return: An offset, data tuple such as readv yields, for the unparsed
data. (which may length 0).
i i i i N( R" R R
R R} R t
splitlinesR RN R R R R RP R RQ R t
_parsed_bytesR R R R~ ( R
t bytesR RX R t
header_end( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyt _parse_header_from_bytes s>
: c C@ sy | t | } | } x\ t rt | j | } | | j | d k rI d S| j | | | | \ } } | r d Sq Wd S( s Parse node data returned from a readv operation.
:param offset: The byte offset the data starts at.
:param data: The data to parse.
i N( R" R R R t _parse_segment( R
R t dataR t high_parsedR t last_segment( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyt
_parse_region s c C@ s d } | | j | d k r; | j | d | } t } n2 | | j | d k ra d } t } n d } t } | | j k r d } t } t } n | d t | j k r d } t } t } n | | j | d d k r d } t } t } nj | | j | d d k rG| j | d d | } t } | | j | d d k } n d } t } t } | s| d k r| j d d } n | j d | d } | d k st d qn | s| d k r| j d d } n | j d d | d } | d k st d qn | | | !}
|
sPt d | | | | t | f n | rc| | 7} n |
j d } | d =| } | j
| | \ }
} } } x! | D] \ } } | | j | <qW| j | |
| t |
| | t |
| f S( sY Parse one segment of data.
:param offset: Where 'data' begins in the file.
:param data: Some data to parse a segment of.
:param end: Where data ends
:param index: The current index into the parsed bytes map.
:return: True if the parsed segment is the last possible one in the
range of data.
:return: high_parsed_byte, last_segment.
high_parsed_byte is the location of the highest parsed byte in this
segment, last_segment is True if the parsed segment is the last
possible one in the data block.
i i s
s no
was presents' read unneeded data [%d:%d] from [%d:%d]iN(
R R R R R R" t findt AssertionErrort rfindR R R~ R ( R
R R R R t trim_endt
trim_startt start_adjacentt end_adjacentR t trimmed_dataRX R t first_keyt last_keyR[ Rt R R2 ( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyR sn
#
c C@ s d } d } d } g } x| D]} | d k r| | j rl | j | d k sl t d | j | f ql n | d 7} q n | j d } t | | j k r t j | n t g | | j D] } t
| ^ q } | d k r | } n | d \ }
} } g }
xR | j d D]A } |
j t g | j d D] } | r;t | ^ q; q Wt |
}
| |
|
| f | j
| <| t | d 7} |
rq n | j r| |
f } n | } | j | | f q W| | | | f S( Ni RE i s %s %sRL is s
( R R R R R" R R
R R R R= R< R R R ( R
RX R R R R R[ t lineR R% R0 R1 R2 Ru t
ref_stringRB R ( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyR T s@
, 6 c C@ s, | j | } | | f } | | f } | d k r] | j j | | | j j | | d S| d t | j k r!| j | d | k r!| j | d d | k r!| j | d | j | d d f | j | <| j | d | j | d d f | j | <| j | d =| j | d =n| j | d | k rw| j | d | f | j | <| j | d | f | j | <n | d t | j k r| j | d d | k r| | j | d d f | j | d <| | j | d d f | j | d <n. | j j | d | | j j | d | d S( s Mark the bytes from start to end as parsed.
Calling self._parsed_bytes(1,2) will mark one byte (the one at offset
1) as parsed.
:param start: The start of the parsed region.
:param end: The end of the parsed region.
iNi i ( R R t insertR R" ( R
R t start_keyR t end_keyR t new_valuet new_key( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyR | s2 !&)c C@ s | s
d S| j d k r= | j d | j k r= | j d S| j } | d k r~ g | D] \ } } | | | f ^ qY } n | j j | j | t | j | j } x | D] \ } } | | 8} | j t
| 7_ | d k r | | } d } n | d k r0t
| | j k r0| j t | d S| j d k ro| d k sWt
n | j | \ } } n | j | | q Wd S( sq Read the ranges and parse the resulting data.
:param readv_ranges: A prepared readv range list.
Ni i ( R R R R R R R| t readvR} R R" R R~ R R R ( R
R t base_offsetR R t
readv_dataR R ( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyR s2 %
)
! c C@ s t S( s' The file signature for this index type.( RM ( R
( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyR s c C@ s x | j D] } q
Wd S( s6 Validate that everything in the index can be accessed.N( R' ( R
R+ ( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyt validate s N(" Ry Rz R{ R R R R R R R RI R R5 R' R R R R R R R Rm R R RY R R R R R R R R R ( ( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyR x s< & .
N , l ( 2 + c B@ s e Z d Z d d Z d Z d Z d Z e Z d d Z
d Z d Z d Z
d Z d
Z d Z d Z d
Z e Z d Z d Z d Z RS( s A GraphIndex made up from smaller GraphIndices.
The backing indices must implement GraphIndex, and are presumed to be
static data.
Queries against the combined index will be made against the first index,
and then the second and so on. The order of indices can thus influence
performance significantly. For example, if one index is on local disk and a
second on a remote server, the local disk index should be before the other
in the index list.
Also, queries tend to need results from the same indices as previous
queries. So the indices will be reordered after every query to put the
indices that had the result(s) of that query first (while otherwise
preserving the relative ordering).
c C@ s8 | | _ | | _ g | _ d g t | j | _ d S( sN Create a CombinedGraphIndex backed by indices.
:param indices: An ordered list of indices to query for data.
:param reload_func: A function to call if we find we are missing an
index. Should have the form reload_func() => True/False to indicate
if reloading actually changed anything.
N( t _indicest _reload_funct _sibling_indicesR R" t _index_names( R
t indicest reload_func( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyR s c C@ s) d | j j d j t t | j f S( Ns %s(%s)s , ( R Ry RU t mapt reprR ( R
( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyR s c C@ s" x | j D] } | j q
Wd S( s See GraphIndex.clear_cache()N( R RI ( R
R ( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyRI
s c C@ s t | } t j | k r> | j t j i g t j 6} n i } xL | j | D]; \ } } } } | d } | s t j f } n | | | <qT W| S( s/ See graph.StackedParentsProvider.get_parent_mapi ( R t
_mod_revisiont
NULL_REVISIONRG Rm ( R
R R t
found_parentsR R R2 R* t parents( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyR s "
c C@ s* | j j | | | j j | | d S( s Insert a new index in the list of indices to query.
:param pos: The position to insert the index.
:param index: The index to insert.
:param name: a name for this index, e.g. a pack name. These names can
be used to reflect index reorderings to related CombinedGraphIndex
instances that use the same names. (see set_sibling_indices)
N( R R R ( R
R R R ( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyt insert_index# s c c@ s t } x t r y\ xQ | j D]F } x= | j D]/ } | d | k r2 | V| j | d q2 q2 Wq Wd SWq t j k
r | j q Xq Wd S( sn Iterate over all keys within the index
Duplicate keys across child indices are presumed to have the same
value and are only reported once.
:return: An iterable of (index, key, reference_lists, value).
There is no defined order for the result iteration - it will be in
the most efficient order for the index.
i N( R R R R' R( R
t
NoSuchFilet _reload_or_raise( R
t seen_keysR R+ ( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyR' / s
c c@ s t | } g } x t r yu xm | j D]b } | s8 Pn t } x3 | j | D]" } | j | d | Vt } qN W| r( | j | q( q( WPWq t j k
r | j q Xq W| j
| d S( s Iterate over keys within the index.
Duplicate keys across child indices are presumed to have the same
value and are only reported once.
:param keys: An iterable providing the keys to be retrieved.
:return: An iterable of (index, key, reference_lists, value). There is
no defined order for the result iteration - it will be in the most
efficient order for the index.
i N( R R R R Rm t removeR< R
R R t _move_to_front( R
R t hit_indicesR t index_hitR+ ( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyRm E s$
c c@ s t | } | s d St } g } x t r y xy | j D]n } t } xI | j | D]8 } | d | k rs qW n | j | d | Vt } qW W| r; | j | q; q; WPWq( t j k
r | j q( Xq( W| j
| d S( s Iterate over keys within the index using prefix matching.
Duplicate keys across child indices are presumed to have the same
value and are only reported once.
Prefix matching is applied within the tuple of a key, not to within
the bytestring of each key element. e.g. if you have the keys ('foo',
'bar'), ('foobar', 'gam') and do a prefix search for ('foo', None) then
only the former key is returned.
:param keys: An iterable providing the key prefixes to be retrieved.
Each key prefix takes the form of a tuple the length of a key, but
with the last N elements 'None' rather than a regular bytestring.
The first element cannot be 'None'.
:return: An iterable as per iter_all_entries, but restricted to the
keys with a matching prefix to those supplied. No additional keys
will be returned, and every match that is in the index will be
returned.
Ni ( R R R R R R( R< R
R R R ( R
R R R R R R+ ( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyR c s*
c C@ sQ | j t | | k r d S| j | } x | j D] } | j | q6 Wd S( s Rearrange self._indices so that hit_indices are first.
Order is maintained as much as possible, e.g. the first unhit index
will be the first index in _indices after the hit_indices, and the
hit_indices will be present in exactly the order they are passed to
_move_to_front.
_move_to_front propagates to all objects in self._sibling_indices by
calling _move_to_front_by_name.
N( R R" t _move_to_front_by_indexR t _move_to_front_by_name( R
R t hit_namest sibling_idx( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyR s
c
C@ sG t | j | j } d t j k r: t j d | | n g } g } g } g } x t | D] \ } \ } } | | k r | j | | j | t | t | k r | j
| j | d | j
| j | d Pq q_ | j | | j | q_ W| | | _ | | | _ d t j k rCt j d | j n | S( sx Core logic for _move_to_front.
Returns a list of names corresponding to the hit_indices param.
R s9 CombinedGraphIndex reordering: currently %r, promoting %ri s CombinedGraphIndex reordered: %r( t zipR R R R R R t enumerateR< R" R (
R
R t indices_infoR t unhit_namest new_hit_indicest
unhit_indicesR R t idx( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyR s.
c C@ s_ t | j | j } g } x0 | D]( \ } } | | k r" | j | q" q" W| j | d S( sq Moves indices named by 'hit_names' to front of the search order, as
described in _move_to_front.
N( R
R R R< R ( R
R R R R R ( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyR
s c
C@ s t } i } t | } d } x | r| d 7} d } x t | j D] \ } } t }
| } d } x, | r | d 7} | j | | | |
} qn W|
} | d k r t |
} n
| j |
| sJ PqJ qJ W| d k r | j | d } q$ | j | | j | q$ W| | f S( se Find the complete ancestry for the given set of keys.
Note that this is a whole-ancestry request, so it should be used
sparingly.
:param keys: An iterable of keys to look for
:param ref_list_num: The reference list which references the parents
we care about.
:return: (parent_map, missing_keys)
i i N( R R R R R t intersection_updateR) t difference_update(
R
R Ro Rr Rq t keys_to_lookupt
generationt all_index_missingt index_idxR t index_missing_keysR t sub_generation( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyRx s6
c C@ sP xI t rK y t d | j D d SWq t j k
rG | j q Xq Wd S( sG Return an estimate of the number of keys in this index.
For CombinedGraphIndex this is approximated by the sum of the keys of
the child indices. As child indices may have duplicate keys this can
have a maximum error of the number of child indices * largest number of
keys in any index.
c s@ s | ] } | j Vq d S( N( RY ( RJ R ( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pys <genexpr> s i N( R RR R R
R R ( R
( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyRY s
c C@ sf | j d k r n t j \ } } } t j d | | j sb t j d | | | n d S( s We just got a NoSuchFile exception.
Try to reload the indices, if it fails, just raise the current
exception.
s, Trying to reload after getting exception: %ssG _reload_func indicated nothing has changed. Raising original exception.N( R R t syst exc_infoR R ( R
t exc_typet exc_valuet
exc_traceback( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyR s
c C@ s
| | _ d S( sM Set the CombinedGraphIndex objects to reorder after reordering self.
N( R ( R
t sibling_combined_graph_indices( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyt set_sibling_indices0 s c C@ sX xQ t rS y& x | j D] } | j q Wd SWq t j k
rO | j q Xq Wd S( s6 Validate that everything in the index can be accessed.N( R R R R
R R ( R
R ( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyR 5 s N( Ry Rz R{ R R R RI R R t has_keyR R' Rm R R R R
Rx RY R Rr R R" R ( ( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyR s& * "
@ c B@ sD e Z d Z d Z d Z d Z d Z d Z d Z RS( s A GraphIndex which operates entirely out of memory and is mutable.
This is designed to allow the accumulation of GraphIndex entries during a
single write operation, where the accumulated entries need to be immediately
available - for example via a CombinedGraphIndex.
c C@ sd | j r9 xT | D]" \ } } } | j | | | q Wn' x$ | D] \ } } | j | | q@ Wd S( sn Add nodes to the index.
:param nodes: An iterable of (key, node_refs, value) entries to add.
N( R RH ( R
R[ R R2 R6 ( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyt add_nodesH s
c c@ s d t j k r" t j d d n | j rq x | j j D]/ \ } \ } } } | s; | | | | f Vq; q; Wn@ x= | j j D], \ } \ } } } | s | | | f Vq q Wd S( s Iterate over all keys within the index
:return: An iterable of (index, key, reference_lists, value). There is no
defined order for the result iteration - it will be in the most
efficient order for the index (in this case dictionary hash order).
R i s- iter_all_entries scales with size of history.N( R R R R R R R- ( R
R R0 R1 R2 ( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyR' T s
%%c c@ s | j } g | D] } | | k r | ^ q } | j r{ x{ | D]6 } | | } | d s> | | | d | d f Vq> q> Wn: x7 | D]/ } | | } | d s | | | d f Vq q Wd S( s_ Iterate over keys within the index.
:param keys: An iterable providing the keys to be retrieved.
:return: An iterable of (index, key, value, reference_lists). There is no
defined order for the result iteration - it will be in the most
efficient order for the index (keys iteration order in this case).
i i i N( R R ( R
R R[ R R+ ( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyRm g s
%
#
c
c@ sO t | } | s d S| j d k r x | D] } | d d k rT t j | n t | | j k r{ t j | n | j | } | d r q, n | j r | | | d | d f Vq, | | | d f Vq, Wd S| j } xe| D]]} | d d k rt j | n t | | j k r9t j | n | } t | } yB x; t | r| d d k r| | d } | j
d qQWWn t k
rq n Xt | r;| g } x | r7| j
d } | j j
\ } } t | t k r| j | j qx | j D] } | f | Vq WqWq | f | Vq Wd S( sl Iterate over keys within the index using prefix matching.
Prefix matching is applied within the tuple of a key, not to within
the bytestring of each key element. e.g. if you have the keys ('foo',
'bar'), ('foobar', 'gam') and do a prefix search for ('foo', None) then
only the former key is returned.
:param keys: An iterable providing the key prefixes to be retrieved.
Each key prefix takes the form of a tuple the length of a key, but
with the last N elements 'None' rather than a regular bytestring.
The first element cannot be 'None'.
:return: An iterable as per iter_all_entries, but restricted to the
keys with a matching prefix to those supplied. No additional keys
will be returned, and every match that is in the index will be
returned.
Ni i i i( R R R R
R! R" R R R5 R R R R- R R R R R (
R
R R R+ R/ R3 R R R R2 ( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyR ~ sP
c C@ s t | j t | j S( sw Return an estimate of the number of keys in this index.
For InMemoryGraphIndex the estimate is exact.
( R" R R ( R
( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyRY s c C@ s d S( s9 In memory index's have no known corruption at the moment.N( ( R
( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyR s ( Ry Rz R{ R$ R' Rm R RY R ( ( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyR @ s G c B@ se e Z d Z d
d Z d Z d d Z d Z d Z d Z d Z
d Z d Z RS( s An adapter between GraphIndex with different key lengths.
Queries against this will emit queries against the adapted Graph with the
prefix added, queries for all items use iter_entries_prefix. The returned
nodes will have their keys and node references adjusted to remove the
prefix. Finally, an add_nodes_callback can be supplied - when called the
nodes and references being added will have prefix prepended.
c C@ s? | | _ | d | | _ | | _ t | | _ | | _ d S( s1 Construct an adapter against adapted with prefix.N( N( t adaptedR t
prefix_keyt prefixR" t
prefix_lent add_nodes_callback( R
R% R' t missing_key_lengthR) ( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyR s
c @ s t | } g } yW xP | D]H \ } } } t f d | D } | j j | | | f q WWnB t k
r x2 | D]& \ } } | j j | | f q Wn X j | d S( sn Add nodes to the index.
:param nodes: An iterable of (key, node_refs, value) entries to add.
c 3@ s+ | ]! } t f d | D Vq d S( c 3@ s | ] } j | Vq d S( N( R' ( RJ t ref_node( R
( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pys <genexpr> s N( R ( RJ R` ( R
( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pys <genexpr> s N( R R< R' R R) ( R
R[ t translated_nodesR R2 R6 t adjusted_references( ( R
s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyR$ s
"c C@ s | j | | | f f d S( s Add a node to the index.
:param key: The key. keys are non-empty tuples containing
as many whitespace-free utf8 bytestrings as the key length
defined for this index.
:param references: An iterable of iterables of keys. Each is a
reference to another key.
:param value: The value to associate with the key. It may be any
bytes as long as it does not contain or
.
N( R$ ( R
R R2 R1 ( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyRH s c #@ s x | D] } | d j j k r9 t j n xK | d D]? } x6 | D]. } | j j k rQ t j qQ qQ WqD W| d | d j | d t f d | d D f Vq Wd S( s+ Strip prefix data from nodes and return it.i i i i c 3@ s+ | ]! } t f d | D Vq d S( c 3@ s | ] } | j Vq d S( N( R( ( RJ R+ ( R
( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pys <genexpr> s N( R ( RJ R` ( R
( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pys <genexpr> s N( R( R' R
R R ( R
t an_iterR+ R` R+ ( ( R
s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyt
_strip_prefix s
c C@ s | j | j j | j g S( s Iterate over all keys within the index
iter_all_entries is implemented against the adapted index using
iter_entries_prefix.
:return: An iterable of (index, key, reference_lists, value). There is no
defined order for the result iteration - it will be in the most
efficient order for the index (in this case dictionary hash order).
( R/ R% R R& ( R
( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyR' s
c @ s) j j j f d | D S( s_ Iterate over keys within the index.
:param keys: An iterable providing the keys to be retrieved.
:return: An iterable of (index, key, value, reference_lists). There is no
defined order for the result iteration - it will be in the most
efficient order for the index (keys iteration order in this case).
c 3@ s | ] } j | Vq d S( N( R' ( RJ R ( R
( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pys <genexpr>, s ( R/ R% Rm ( R
R ( ( R
s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyRm # s c @ s) j j j f d | D S( sl Iterate over keys within the index using prefix matching.
Prefix matching is applied within the tuple of a key, not to within
the bytestring of each key element. e.g. if you have the keys ('foo',
'bar'), ('foobar', 'gam') and do a prefix search for ('foo', None) then
only the former key is returned.
:param keys: An iterable providing the key prefixes to be retrieved.
Each key prefix takes the form of a tuple the length of a key, but
with the last N elements 'None' rather than a regular bytestring.
The first element cannot be 'None'.
:return: An iterable as per iter_all_entries, but restricted to the
keys with a matching prefix to those supplied. No additional keys
will be returned, and every match that is in the index will be
returned.
c 3@ s | ] } j | Vq d S( N( R' ( RJ R ( R
( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pys <genexpr>@ s ( R/ R% R ( R
R ( ( R
s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyR . s c C@ s t t | j S( s Return an estimate of the number of keys in this index.
For GraphIndexPrefixAdapter this is relatively expensive - key
iteration with the prefix is done.
( R" R R' ( R
( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyRY B s c C@ s | j j d S( s Call the adapted's validate.N( R% R ( R
( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyR J s N( (
Ry Rz R{ R R R$ RH R/ R' Rm R RY R ( ( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyR s
(" R{ t
__future__R t __all__t bisectR t cStringIOR t reR t bzrlib.lazy_importR t globalst bzrlibR R
t bzrlib.static_tupleR R RP RQ RN RM t compileR# R: R R t objectR R R R R ( ( ( s0 /usr/lib/python2.7/dist-packages/bzrlib/index.pyt <module> sD 1 o ]
Copyright © 2017 || Recoded By Mr.Bumblebee