-=[ Mr. Bumblebee ]=-
_Indonesia_
:Nc @@ s% d Z d d l m Z d d l Z d d l Z d d l m Z e j e d d d l m Z m Z m
Z
m Z m Z m
Z
d d l m Z d d d Z e j Z e e _ d
Z d Z d Z d
Z d Z e j Z e j d e d e f d YZ d e f d YZ e Z d e f d YZ! d e f d YZ" d Z# d e f d YZ$ e d Z% y, d d l& m' Z' m( Z( m) Z) m* Z* m+ Z+ WnH e, k
rZ- e
j. e- d d l/ m' Z' m( Z( m) Z) m* Z* m+ Z+ n Xe j d e( e j d e) d Z0 d S( s} Persistent maps from tuple_of_strings->string using CHK stores.
Overview and current status:
The CHKMap class implements a dict from tuple_of_strings->string by using a trie
with internal nodes of 8-bit fan out; The key tuples are mapped to strings by
joining them by , and padding shorter keys out to the length of the
longest key. Leaf nodes are packed as densely as possible, and internal nodes
are all an additional 8-bits wide leading to a sparse upper tree.
Updates to a CHKMap are done preferentially via the apply_delta method, to
allow optimisation of the update operation; but individual map/unmap calls are
possible and supported. Individual changes via map/unmap are buffered in memory
until the _save method is called to force serialisation of the tree.
apply_delta records its changes immediately by performing an implicit _save.
TODO:
-----
Densely packed upper nodes.
i ( t absolute_importN( t lazy_imports(
from bzrlib import (
errors,
)
( t errorst lru_cachet osutilst registryt static_tuplet trace( t StaticTuplei i c C@ s= t t d d } | d k r9 t j t } | t _ n | S( s Get the per-thread page cache.
We need a function to do this because in a new thread the _thread_caches
threading.local object does not have the cache initialized yet.
t
page_cacheN( t getattrt _thread_cachest NoneR t LRUSizeCachet _PAGE_CACHE_SIZER ( R ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyt
_get_cacheI s
c C@ s t j d S( N( R t clear( ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyt clear_cacheW s i2 i c C@ s
d j | S( sD Map the key tuple into a search string that just uses the key bytes.t ( t join( t key( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyt _search_key_plaina s t plaint CHKMapc B@ s e Z d Z d Z d d Z d Z d Z d Z d Z e
d Z e d
Z
e d d d d
Z e d d d d Z e d d d d Z d Z d d Z d Z d Z d Z d Z e d Z d Z d Z RS( s= A persistent map from string to string backed by a CHK store.t _storet
_root_nodet _search_key_funcc C@ s^ | | _ | d k r t } n | | _ | d k rH t d | | _ n | j | | _ d S( so Create a CHKMap object.
:param store: The store the CHKMap is stored in.
:param root_key: The root key of the map. None to create an empty
CHKMap.
:param search_key_func: A function mapping a key => bytes. These bytes
are then used by the internal nodes to split up leaf nodes into
multiple pages.
t search_key_funcN( R R R R t LeafNodeR t _node_key( t selft storet root_keyR ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyt __init__o s
c
C@ s0 t } t j } t g | D]3 \ } } } | d k r | d k r | | ^ q } t | j d | } | r t j | d | n xK | D]C \ } } } | d k r | | k r | j | d t t
} q q Wx6 | D]. \ } } } | d k r | j | | q q W| r&| j n | j
S( s Apply a delta to the map.
:param delta: An iterable of old_key, new_key, new_value tuples.
If new_key is not None, then new_key->new_value is inserted
into the map; if old_key is not None, then the old mapping
of old_key is removed.
t
key_filters$ New items are already in the map %r.t check_remapN( t FalseR t
from_sequencet setR t listt iteritemsR t InconsistentDeltaDeltat unmapt Truet mapt _check_remapt _save(
R t deltat has_deletest as_stt oldR t valuet new_itemst existing_newt new( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyt apply_delta s$ -
c C@ s1 t | j t k r- | j | j | _ n d S( s1 Ensure that the root node is an object not a key.N( t typeR R t _get_node( R ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyt _ensure_root s c C@ s? t | t k r7 | j | } t | | d | j S| Sd S( s Get a node.
Note that this does not update the _items dict in objects containing a
reference to this node. As such it does not prevent subsequent IO being
performed.
:param node: A tuple key or node object.
:return: A node object.
R N( R8 R t _read_bytest _deserialiseR ( R t nodet bytes( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyR9 s
c C@ se y t | SWnO t k
r` | j j | g d t } | j j d } | t | <| SXd S( Nt unorderedt fulltext( R t KeyErrorR t get_record_streamR+ t nextt get_bytes_as( R R t streamR> ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyR; s
c C@ sH | j | j | j d d d d d | } | j d d j | S( s+ Return the tree in a string representation.t prefixt t indentt include_keyss
( R: t _dump_tree_nodeR t appendR ( R RI t res( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyt
_dump_tree s
c C@ s( g } | s d } n2 | j } | d k rA d | d f } n d } | j d | | | j j | f t | t k r t | j | j x t
| j j D]2 \ } } | j
| j | | | d d | q WnC x@ t
| j j D]) \ } }
| j d t | |
f q W| S(
sA For this node and all children, generate a string representation.RG s %si s Nones %s%r %s%ss RI s %r %rN( R R RK t __class__t __name__R8 t InternalNodeR' t _iter_nodesR t sortedt _itemsR( t extendRJ t tuple( R R= RF RH RI t resultt key_strt node_keyt subR R3 ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyRJ s" " "!i i c C@ sV | j | | d | d | d | } t | t k rR t d t | f n | S( s5 Create a CHKMap in store with initial_value as the content.
:param store: The store to record initial_value in, a VersionedFiles
object with 1-tuple keys supporting CHK key generation.
:param initial_value: A dict to store in store. Its keys and values
must be bytestrings.
:param maximum_size: The maximum_size rule to apply to nodes. This
determines the size at which no new data is added to a single node.
:param key_width: The number of elements in each key_tuple being stored
in this map.
:param search_key_func: A function mapping a key => bytes. These bytes
are then used by the internal nodes to split up leaf nodes into
multiple pages.
:return: The root chk of the resulting CHKMap.
t maximum_sizet key_widthR s$ we got a %s instead of a StaticTuple( t _create_directlyR8 R t AssertionError( t klassR t
initial_valueRZ R[ R R ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyt from_dict s c C@ s} | | d d | } | j j | | | j _ g } x0 | j D]" \ } } | j d | | f qD W| j | }
|
S( NR ( R R t set_maximum_sizet
_key_widtht itemsRK R7 ( R^ R R_ RZ R[ R RV R/ R R3 R ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyt _create_via_map s c C@ s t d | } | j | | | _ t j } t g | j D] \ } } | | | f ^ q> | _ t g | j j D] \ } }
| j | |
^ q{ | _
t | j | _ | j
| j | j d k rz| rz| j | k rz| j | \ } } t | d k r(t d n t | d | } | j | | | _ x' | D] \ }
} | j |
| qWWn t | j | } | d S( NR i s! Failed to split using node._spliti( R Ra Rb R R% t dictR( RS t sumt _key_value_lent _raw_sizet lent _lent _compute_search_prefixt _compute_serialised_prefixt
_current_sizet _splitR] RP t add_nodeR' t serialise( R^ R R_ RZ R[ R R= R1 R t valR3 RF t node_detailst splitt subnodet keys( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyR\ s0
47
c #@ sN j j j j k r( d S j j t j } j } g g d f d f d f d } | | d | d t } t } t f d } d } xM s rI| d 7} sx D] \ } }
} } | | rDq n j | } |
d k rp|
| d f Vq x. | j j D] \ }
}
|
|
d f VqWq Wd S s?x D] \ } }
} } | | rqn j | } |
d k r|
d | f Vqx. | j j D] \ }
}
|
d |
f VqWqWd S d d d d k rt j \ } }
} } | | rq n |
d k r|
d | f VqF | | q q d d d d k r=t j \ } }
} } | | rq n |
d k r$|
| d f VqF | | q q d d d k rZt
} n t } d d d k r}t
} n t } | r| rt j } t j } | d | d k r | d | d | d f Vq q n j d d j d d k r4t j t j q n | r| rt j \ } } } } t j \ } } } } | | k rt d
| | f n | | | | | q n | rt j \ } }
} } | | rq n | | n | r t j \ } }
} } | | r0q n | | q q Wd S( s Iterate over the changes between basis and self.
:return: An iterator of tuples: (key, old_value, new_value). Old_value
is None for keys only in self; new_value is None for keys only in
basis.
Nc S@ s | j | } t | t k r~ | j | f } x | j j D]7 \ } } | j | } t j | | | | | f q@ WnK | j | f } x9 | j j D]( \ } } t j | | d | | f q Wd S( N(
R9 R8 R t _keyRS Rc R t heapqt heappushR ( R= t patht a_mapt pendingR R3 t
search_keyRF t child( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyt process_nodeK s # c @ s t | j j } t | j j } | j d f } x4 | | D]( \ } } t j | d | | f qD W| j d f } x4 | | D]( \ } } t j | d | | f q Wd S( N( R& RS Rc Rv R Rw Rx ( t self_nodet
basis_nodet
self_itemst basis_itemsRy RF R} ( t
basis_pendingt self_pending( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyt process_common_internal_nodesZ s c @ s t | j j } t | j j } | j d f } xC | | D]7 \ } } j | } t j | | | | f qD W| j d f } xC | | D]7 \ } } j | } t j | | | | f q Wd S( N( R& RS Rc Rv R R Rw Rx ( R R R R Ry R R3 RF ( t basisR R R ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyt process_common_leaf_nodesc s c @ s j | } j | } t | t k rR t | t k rR | | nZ t | t k r t | t k r | | n& | | | | d S( N( R9 R8 RP R ( R t self_pathR t
basis_path( R R R R R~ R R ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyt process_common_prefix_nodesn s c @ s3 x, | d k r. | \ } } | k r t Sq Wt S( N( R R+ R$ ( t key_pathR ( t
excluded_keys( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyt check_excluded s
i i i s %r != %r(
R R R: R& R R9 R( R Rw t heappopR+ R$ R] ( R R R R R t self_seent
basis_seenR t loop_counterRF R R= Ry R3 t read_selft
read_basist self_detailst
basis_detailst self_prefixt _R t basis_prefixR ( ( R R R R R R~ R R s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyt iter_changes# s $
!
c C@ sZ | j | d k rA t j } g | D] } | | ^ q& } n | j j | j d | S( s* Iterate over the entire CHKMap's contents.R" N( R: R R R% R R( R ( R R" R1 R ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyR( s
"c C@ s* t | j t k r | j S| j j Sd S( s Return the key for this map.N( R8 R R Rv ( R ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyR s c C@ s | j t | j S( N( R: Ri R ( R ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyt __len__ s
c C@ s t j | } | j | j j | j | | \ } } t | d k r` | d d | _ nt t | d | j | _ | j j | d d j
| d d j | j _ x' | D] \ } } | j j | | q Wd S( s Map a key tuple to value.
:param key: A key to map.
:param value: The value to assign to key.
i i R N(
R R% R: R R, R Ri RP R Ra RZ Rb Ro ( R R R3 RF Rr Rs R= ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyR, s
! c C@ sE t | t k r$ t j | } n t | t k r: | S| j Sd S( s4 Get the key for a node whether it's a tuple or node.N( R8 RU R R% Rv ( R R= ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyR ' s
c C@ st t j | } | j t | j t k rO | j j | j | d | } n | j j | j | } | | _ d S( s remove key from the map.R# N( R R% R: R8 R RP R* R ( R R R# t unmapped( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyR* 0 s
c C@ s> | j t | j t k r: | j j | j | _ n d S( s Check if nodes can be collapsed.N( R: R8 R RP R- R ( R ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyR- ; s
c C@ s? t | j t k r | j St | j j | j } | d S( sM Save the map completely.
:return: The key of the root node.
i( R8 R R R' Rp R ( R Ru ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyR. A s ( s _stores
_root_nodes _search_key_funcN( RO t
__module__t __doc__t __slots__R R! R7 R: R9 R; R$ RM R+ RJ t classmethodR` Rd R\ R R( R R R, R R* R- R. ( ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyR j s6
t Nodec B@ sq e Z d Z d Z d d
Z d Z d Z d
Z e d Z d Z
e d Z e d Z
RS( s Base class defining the protocol for CHK Map nodes.
:ivar _raw_size: The total size of the serialized key:value data, before
adding the header bytes, and without prefix compression.
Rv Rj t
_maximum_sizeRb Rh RS t _search_prefixR i c C@ sC d | _ d | _ d | _ | | _ d | _ i | _ d | _ d S( sS Create a node.
:param key_width: The width of keys for this node.
i N( R Rv Rj R Rb Rh RS R ( R R[ ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyR! X s c C@ sj t t | j } t | d k r8 | d d } n d | j j | j | j | j | j | j
| f S( Ni i s ...]s3 %s(key:%s len:%s size:%s max:%s prefix:%s items:%s)( t strRR RS Ri RN RO Rv Rj Rh R R ( R t items_str( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyt __repr__i s c C@ s | j S( N( Rv ( R ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyR q s c C@ s | j S( N( Rj ( R ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyR t s c C@ s | j S( s8 What is the upper limit for adding references to a node.( R ( R ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyRZ w s c C@ s
| | _ d S( s Set the size threshold for nodes.
:param new_size: The size at which no data is added to a node. 0 for
unlimited.
N( R ( R t new_size( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyRa | s c C@ sq | j | r | Sd } xC t t | | D], \ } \ } } | | k r/ | d 8} Pq/ q/ W| | d } | S( s Given 2 strings, return the longest prefix common to both.
:param prefix: This has been the common prefix for other keys, so it is
more likely to be the common prefix in this case as well.
:param key: Another string to compare to
ii ( t
startswitht enumeratet zip( t clsRF R t post leftt rightt common( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyt
common_prefix s (
c C@ sO d } xB | D]: } | d k r+ | } q
n | j | | } | s
d Sq
W| S( s Given a list of keys, find their common prefix.
:param keys: An iterable of strings.
:return: The longest common prefix of all keys.
RG N( R R ( R Ru R R ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyt common_prefix_for_keys s
( s _keys _lens
_maximum_sizes
_key_widths _raw_sizes _itemss _search_prefixs _search_key_func( RO R R R R! R R R t propertyRZ Ra R R R ( ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyR M s R c B@ s e Z d Z d Z d d Z d Z d Z e d d Z d d Z
d Z d Z d Z
d
Z d Z d j Z d
Z d Z d Z d Z d Z d Z RS( s A node containing actual key:value pairs.
:ivar _items: A dict of key->value items. The key is in tuple form.
:ivar _size: The number of bytes that would be used by serializing all of
the key/value pairs.
t _common_serialised_prefixc C@ s; t j | d | _ | d k r. t | _ n | | _ d S( N( R R! R R R R ( R R ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyR! s
c C@ sp t t | j } t | d k r8 | d d } n d | j j | j | j | j | j | j
| j | f S( Ni i s ...]s? %s(key:%s len:%s size:%s max:%s prefix:%s keywidth:%s items:%s)( R RR RS Ri RN RO Rv Rj Rh R R Rb ( R R ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyR s c C@ s | j d k r d } d } n# t | j } | j | | j } d t t | j d t t | j d t t | j d | d | S( s Answer the current serialised size of this node.
This differs from self._raw_size in that it includes the bytes used for
the header.
i i i N( R R Ri Rh Rj R R Rb ( R t bytes_for_itemst
prefix_len( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyRm s c C@ s" t j | } t | | d | S( s Deserialise bytes, with key key, into a LeafNode.
:param bytes: The bytes of the node.
:param key: The key that the serialised node has.
R ( R t expect_static_tuplet _deserialise_leaf_node( R^ R> R R ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyt deserialise s c c@ s | d k r i } xx | D]p } t | | j k ra y | | j | f VWq t k
r] q Xq | j t | t } | j | q W| r| j } xN | j j D]: } x1 | D]) \ } } | d | | k r | VPq q Wq Wqn x | j j D] } | VqWd S( s Iterate over items in the node.
:param key_filter: A filter to apply to the node. It should be a
list/set/dict or similar repeatedly iterable container.
i N(
R Ri Rb RS RA t
setdefaultR& t addRc R( ( R R R" t filtersR t
length_filtert itemt length( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyR( s&
c C@ sB t | j | d t t | j d d t | d S( Ni s
( Ri t _serialise_keyR t count( R R R3 ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyRg s c C@ s
| j | S( N( R ( R R ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyt _search_key s c C@ s+ | | j | <| j | j | | 7_ | j d 7_ | j | } | j d k ra | | _ n | j | j | | _ | j | } | j t
k r | j n | j d k r | | _ n | j | j | | _ | j d k r'| j r'| j
| j k r'| | j k s | j r't Sn t S( s Map a key to a value.
This assumes either the key does not already exist, or you have already
removed its size and length from self.
:return: True if adding this node should cause us to split.
i N( RS Rh Rg Rj R R R R R R t _unknownRk R Rm t _are_search_keys_identicalR+ R$ ( R R R3 t serialised_keyR| ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyt
_map_no_split s,
c C@ s | j t k r t d n | j } t | d } i } xb| j j D]Q\ } } | j | } | | } t | | k r | d | t | 7} n | | k r t d | j } | j | j
| j | _ | | | <n
| | } | j | | | \ }
} t | d k rM | |
k r9| j
| n t |
d | j } | j | j
| j | _ x$ | D] \ }
} | j |
| qqW| | | <qM qM W| | j f S( s We have overflowed.
Split this node into multiple LeafNodes, return it up the stack so that
the next layer creates a new InternalNode and references the new nodes.
:return: (common_serialised_prefix, [(node_serialised_prefix, node)])
s Search prefix must be knowni R R ( R R R] Ri RS R( R R R Ra R Rb R, t popRP Ro Rc ( R R R t split_atRV R R3 R| RF R= t
sub_prefixRr t new_nodeRs ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyRn < s8
c C@ s | | j k rC | j | j | | j | 8_ | j d 8_ n d | _ | j | | rk | j | S| j t k r t
d | j n | j d | f g f Sd S( s Map key to value.i s %r must be knownRG N( RS Rh Rg Rj R Rv R Rn R R R] ( R R R R3 ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyR, l s "
R c C@ s d g } | j d | j | j d | j | j d | j | j d k r | j d t | j d k r t d q n& | j d | j f t | j } x t | j j
D] \ } } t j | d g } d | j
| t | f } | j | j s3t d | j | f n | j | | | j | q W| j d
d | \ } } } t d | j | _ d
j | }
t |
| j k rt d n |
t | j <| j g S( s Serialise the LeafNode to store.
:param store: A VersionedFiles honouring the CHK extensions.
:return: An iterable of the keys inserted by this operation.
s chkleaf:
s %d
s
i s< If _common_serialised_prefix is None we should have no itemss %s
s %s %s
sK We thought the common prefix was %r but entry %r does not have it in commons sha1:RG s Invalid _current_sizeN( N( ( RK R Rb Rj R R Ri RS R] RR Rc R t chunks_to_linesR R RT t add_linesR t internRv R Rm R ( R R t linesR R R3 t value_linest
serializedt sha1R R> ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyRp { s4
" c C@ s g S( s7 Return the references to other CHK's held by this node.( ( R ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyt refs s c C@ s> g | j D] } | j | ^ q
} | j | | _ | j S( s Determine the common search prefix for all keys in this node.
:return: A bytestring of the longest search key prefix that is
unique within this node.
( RS R R R ( R R t search_keys( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyRk s %c C@ sR d } xE | j D]: } | j | } | d k r: | } q | | k r t Sq Wt S( s Check to see if the search keys for all entries are the same.
When using a hash as the search_key it is possible for non-identical
keys to collide. If that happens enough, we may try overflow a
LeafNode, but as all are collisions, we must not split.
N( R RS R R$ R+ ( R t common_search_keyR R| ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyR s c C@ s> g | j D] } | j | ^ q
} | j | | _ | j S( s Determine the common prefix for serialised keys in this node.
:return: A bytestring of the longest serialised key prefix that is
unique within this node.
( RS R R R ( R R t serialised_keys( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyRl s % c C@ s y& | j | j | | j | 8_ Wn* t k
rR t j d | | j n X| j d 8_ | j | =d | _ | j | j
| S( s Unmap key from the node.s key %s not found in %ri N( Rh Rg RS RA R t mutterRj R Rv Rk Rl ( R R R ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyR* s &
( s _common_serialised_prefixN( RO R R R R R! R Rm R R R( Rg R R Rn R, R R Rp R Rk R Rl R* ( ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyR s(
# $ 0
%
RP c B@ s e Z d Z d Z d d d Z d Z d Z e d d Z d d Z
d d d Z d Z d
Z
d Z d Z d
Z d Z d Z d d Z e d Z d Z RS( s A node that contains references to other nodes.
An InternalNode is responsible for mapping search key prefixes to child
nodes.
:ivar _items: serialised_key => node dictionary. node may be a tuple,
LeafNode or InternalNode.
t _node_widthRG c C@ sD t j | d | _ | | _ | d k r7 t | _ n | | _ d S( Ni ( R R! R R R R R ( R RF R ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyR! s
c C@ s& | j d k r t d n | j | j sL t d | | j f n t | t | j d k r t d | t | j d f n | j t | 7_ t | j s t | | _ n | j t | j d k r t d | j t | j d f n | | j | <d | _ d S( s Add a child node with prefix prefix, and node node.
:param prefix: The search key prefix for node.
:param node: The node being added.
s! _search_prefix should not be Nones( prefixes mismatch: %s must start with %si s& prefix wrong length: len(%s) is not %ds! node width mismatch: %d is not %dN( R R R] R Ri Rj RS R Rv ( R RF R= ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyRo s #
c C@ s@ | j t t | j t t | j t t | j S( s0 Answer the current serialised size of this node.( Rh Ri R Rj Rb R ( R ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyRm s ,c C@ s" t j | } t | | d | S( s Deserialise bytes to an InternalNode, with key key.
:param bytes: The bytes of the node.
:param key: The key that the serialised node has.
:return: An InternalNode instance.
R ( R R t _deserialise_internal_node( R^ R> R R ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyR s c c@ sO xH | j | d | D]1 \ } } x" | j | d | D] } | Vq8 Wq Wd S( NR" ( RQ R( ( R R R" R= t node_filterR ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyR( s " c c@ s i } t } | d k rn t } x | j j D]9 \ } } | j t k r\ | d f | | <q. | d f Vq. Wn t | d k r x | D] } Pq W| j | } t | | j k r t } y | j | } Wn t
k
r d SX| j t k r| | g f | | <q | | g f Vd Sq n | si }
i } x^ | D]V } | j | } | j t | t } | j
| |
j | g j | q8W| j | k r3t | d k r3| | j }
x|
D]e } y | j | } Wn t
k
rqn X|
| } | j t k r!| | f | | <q| | f VqWq| j } x | j j D] \ } } g } x> | D]6 \ } } | | } | | k rh| j |
| qhqhW| rO| j t k r| | f | | <q| | f VqOqOWn | rt } x | D]{ } y t | } Wn t
k
r"qqXt | | d | j } | | \ } } | | j | <| j
| | | f VqWx | D]
} | | =q{Wn | r| d k rt | } n t | } x t d t | | D] } | | | | !} | j | d t } g } x{ | D]s } | j d } t | | j d | j } | | j \ } } | j | | f | | j | <| t | j <qWx | D] } | VqWqWn d S( s` Iterate over node objects which match key_filter.
:param store: A store to use for accessing content.
:param key_filter: A key filter to filter nodes. Only nodes that might
contain a key in key_filter will be returned.
:param batch_size: If not None, then we will return the nodes that had
to be read using get_record_stream in batches, rather than reading
them all at once.
:return: An iterable of nodes. This function does not have to be fully
consumed. (There will be no pending I/O when items are being returned.)
i NR i R? R@ ( R$ R R+ RS R( RN R Ri t _search_prefix_filterR RA R R& R RK Rc RT R R< R R' t rangeRB RD R ( R R R" t
batch_sizeRu t shortcutRF R= R t
search_prefixt prefix_to_keyst length_filtersR t search_prefixest node_key_filterR R t
found_keysR> t key_ordert batch_startt batchRE t node_and_filterst recordt info( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyRQ s
c C@ sF t | j s t d n | j | } | j t | j d k rr t d | j t | j d f n | j | j s | j | j | } t | d | j } | j
| j | j | _ | j
| j t | d | | j | | | Sg | j | d | g D] \ } } | ^ q} | rB| d }
n | j | t }
t |
} t |
t k r|
j } n d } |
j | | | \ }
} t | d k r| d d }
| j | t |
| _ |
| j | <d | _ | } t |
t k r| d k r5t j d | j | } q|
j } | | } | d k rc| t k so| t k rt j d | | | j | } qn | j d k rt d n | j d
| f g f S| j | t }
|
|
_ x$ | D] \ } } |
j
| | qW| j | t |
| _ d | _ | j d
| f g f S( s Map key to value.s# can't map in an empty InternalNode.i s! node width mismatch: %d is not %dR R" i s* checking remap as InternalNode -> LeafNodes, checking remap as size shrunk by %d to be %ds! _search_prefix should not be NoneRG N( Ri RS R] R R R R R RP R Ra R Rb Ro R, RQ t
_new_childR R8 Rm R Rj Rv R R R- t _INTERESTING_NEW_SIZEt _INTERESTING_SHRINKAGE_LIMIT( R R R R3 R| t
new_prefixt
new_parentR= R t childrenR} t old_lent old_sizeRF Rr R R t shrinkageRs ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyR, sl # .
c C@ sB | } | j | j | j | _ | j | _ | | j | <| S( s& Create a new child node of type klass.( Ra R Rb R RS ( R R| R^ R} ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyR s
c
c@ s xc | j j D]R } t | t k r. q n | j d
k rC q n x | j | D] } | VqS Wq Wd g } | j d | j | j d | j | j d | j
| j d
k r t d n | j d | j f t
| j } x t | j j D] \ } } t | t k r0| d } n
| j d } d | | f } | j | j s{t d | | j f n | j | | qW| j d d | \ } } } t d | j | _ d j | t | j <| j Vd
S(
s Serialise the node to store.
:param store: A VersionedFiles honouring the CHK extensions.
:return: An iterable of the keys inserted by this operation.
s chknode:
s %d
s! _search_prefix should not be Nones %s
i s %s %s
s( prefixes mismatch: %s must start with %ss sha1:RG N( N( ( RS t
itervaluesR8 R Rv R Rp RK R Rb Rj R R] Ri RR Rc R R R R R (
R R R= R R R RF t
serialisedR R ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyRp s8
"
c C@ s | j | d | j | j S( s/ Return the serialised key for key in this node.R ( R R ( R R ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyR / s c C@ s | j | | j S( s6 Serialise key for use as a prefix filter in iteritems.( R R ( R R ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyR 5 s c c@ sp | | j k rL x6 | j j D]% } x | j | D] } | Vq5 Wq Wd Sx | j j D] \ } } q\ Wd S( s Split this node into smaller nodes starting at offset.
:param offset: The offset to start the new child nodes at.
:return: An iterable of (prefix, node) tuples. prefix is a byte
prefix for reaching node.
N( R RS t valuesRn Rc ( R t offsetR= RV R ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyRn 9 s
c C@ sw | j d k r t d n g } xL | j j D]; } t | t k r\ | j | q4 | j | j q4 W| S( s7 Return the references to other CHK's held by this node.s unserialised nodes have no refs.N( Rv R R] RS R R8 R RK R ( R R R3 ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyR H s c C@ s | j | j | _ | j S( s Return the unique key prefix for this node.
:return: A bytestring of the longest search key prefix that is
unique within this node.
( R RS R ( R t extra_key( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyRk T s c
C@ s0 t | j s t d n g | j | d | g D] \ } } | ^ q7 } | rb | d } n t | | j d 8_ | j | | } d | _ | j | } t | d k r | j | =d } n
| | j | <t | j d k r | j j
d St | t k r| S| r(| j
| S| Sd S( s+ Remove key from this node and its children.s% can't unmap in an empty InternalNode.R" i i N( Ri RS R] RQ RA Rj R* R Rv R R R8 RP R- (
R R R R# R= R R R} R R| ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyR* ] s, .
c C@ s t d | j } | j | j | j | _ xo | j | d d D]X \ } } t | t k rf | Sx3 | j j D]" \ } } | j
| | rv | Sqv WqD Wt j d | S( s Check if all keys contained by children fit in a single LeafNode.
:param store: A store to use for reading more nodes
:return: Either self, or a new LeafNode which should replace self.
R R i s remap generated a new LeafNode(
R R Ra R Rb RQ R8 RP RS R( R R R ( R R t new_leafR= R R R3 ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyR- | s "
( s _node_widthN( RO R R R R R! Ro Rm R R R( RQ R, R Rp R R Rn R Rk R+ R* R- ( ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyRP s& N & c C@ sd | j d r* t j | | d | } n6 | j d rT t j | | d | } n t d | S( s7 Helper for repositorydetails - convert bytes to a node.s chkleaf:
R s chknode:
s Unknown node type.( R R R RP R] ( R> R R R= ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyR< s t CHKMapDifferencec B@ sb e Z d Z d
d Z d Z d Z d Z d Z d Z d Z
d Z d Z RS( sm Iterate the stored pages and key,value pairs for (new - old).
This class provides a generator over the stored CHK pages and the
(key, value) pairs that are in any of the new maps and not in any of the
old maps.
Note that it may yield chk pages that are common (especially root nodes),
but it won't yield (key,value) pairs that are common.
c C@ s | | _ | | _ | | _ | | _ t | j | _ t | _ t | _ | | _ g | _ g | _
g | _ d | _
d S( N( R t _new_root_keyst _old_root_keyst _pbR& t
_all_old_chkst _all_old_itemst _processed_new_refsR t
_old_queuet
_new_queuet _new_item_queueR t _state( R R t
new_root_keyst
old_root_keysR t pb( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyR! s c c@ s t j } | j j | d t } x | D] } | j d k rM | j j n | j d k rw t j
| j | j n | j d } t
| | j d | j } t | t k r | j j } g } n g } | j j } | | | | f Vq( Wd S( NR? t absentR@ R ( R R% R RB R+ R R t tickt storage_kindR t NoSuchRevisionR RD R< R R8 RP RS Rc ( R Ru R1 RE R R> R= t prefix_refsRc ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyt _read_nodes_from_store s
c C@ s g } | j } x | j | j D] \ } } } } g | D] } | d | k r; | ^ q; } g | D] } | d ^ qd } | j | | j j | | j | q" W| S( Ni ( R R
R t updateR RT ( R t old_chks_to_enqueuet all_old_chksR R= R Rc t p_rt new_refs( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyt _read_old_roots
s "
c C@ sz xs | D]k \ } } t } x: t t | d d D] } | | | k r2 t } Pq2 q2 W| rb q n | j j | q Wd S( Ni i( R+ t xrangeRi R$ R RK ( R t new_prefixesR RF t reft not_interestingt i( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyt _enqueue_old s c c@ s | j s t | j | _ d S| j } t | j j | j } t } | j } | j | x'| j
| D]\ } } } } g | D]/ } | d | j k r | d | k r | ^ q } g | D] } | d ^ q }
| j g | D] } | d ^ q | j j |
g | D] } | | j k r| ^ q} | j
j | | j g | D] } | j | d ^ qY | j |
| Vqu WxG t | D]9 }
| j g t d t |
D] } |
| ^ q qW| j | | d S( s Read the root pages.
This is structured as a generator, so that the root records can be
yielded up to whoever needs them without any buffering.
Ni i ( R R' R R R R& t
differenceR R R R
RT R R R R Ri R ( R R t new_keysR t processed_new_refsR R= R Rc R R R R4 RF R ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyt _read_all_roots. s4
$
$
7c c@ sk t | j } g | _ | j } | j } | j } g | j D] } | | k r= | ^ q= } g | _ | rx d | f Vn | j | } | j | x | rft } | j } x | j | D]o \ } }
} } | r g | D] } | | k r | ^ q } n | | f V| g | D] }
|
d ^ q ~ q W| j | } | j | } | j | | } q Wd S( Ni (
R& R R R R R R R R R
( R R R
R t
all_old_itemsR R4 t next_refst next_refs_updateR R t p_refsRc R ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyt _flush_new_queued s4
"
!
c C@ s | j } g | _ | j } x{ | j | D]j \ } } } } | j j | g | D] \ } } | | k rT | ^ qT } | j j | | j | q+ Wd S( N( R R R
R R RT ( R R R
R R R Rc t r( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyt _process_next_old s "+c C@ s$ x | j r | j q W| j S( N( R R! R ( R ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyt _process_queues s c c@ sN x | j D] } | g f Vq
Wx% | j D] \ } } | | f Vq/ Wd S( N( R R" ( R R Rc ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyt process s N(
RO R R R R! R
R R R R R! R" R# ( ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyR s # " 6 -
c C@ s+ t | | | d | j d | } | j S( s Given root keys, find interesting nodes.
Evaluate nodes referenced by interesting_root_keys. Ones that are also
referenced from uninteresting_root_keys are not considered interesting.
:param interesting_root_keys: keys which should be part of the
"interesting" nodes (which will be yielded)
:param uninteresting_root_keys: keys which should be filtered out of the
result set.
:return: Yield
(interesting record, {interesting key:values})
R R ( R R R# ( R t interesting_root_keyst uninteresting_root_keysR t iterator( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyt iter_interesting_nodes s
( t _bytes_to_text_keyt _search_key_16t _search_key_255R R s hash-16-ways hash-255-wayc C@ s t | t k r1 t d | t | f n t | d k rb t d | t | f n t | d t k r t d | t | d f n | d j d s t d | f n d S( s Helper function to assert that a key is properly formatted.
This generally shouldn't be used in production code, but it can be helpful
to debug problems.
s key %r is not StaticTuple but %si s# key %r should have length 1, not %di s key %r should hold a str, not %rs sha1:s key %r should point to a sha1:N( R8 R t TypeErrorRi t
ValueErrorR R ( R ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyt
_check_key s (1 R t
__future__R Rw t threadingt bzrlibR t globalsR R R R R R t bzrlib.static_tupleR R t localR R R R R R R R t Registryt search_key_registryt registert objectR R R R RP R< R R' t bzrlib._chk_map_pyxR( R) R* R R t ImportErrort et failed_to_load_extensiont bzrlib._chk_map_pyR- ( ( ( s2 /usr/lib/python2.7/dist-packages/bzrlib/chk_map.pyt <module>& sJ . _ , ,
,
Copyright © 2017 || Recoded By Mr.Bumblebee