o >ag:@sdZddlZddlmZddlZddlmZddl m Z ddl m Z ddl m Zddl mZddlZedZeeed d d gZed gd ZGdddZdS)z; Helper functions to communicate with replication servers. N) namedtuple)ceil)MergeInputReader)io)versionpyosmium OsmosisStatesequence timestampDownloadResult)idreadernewestc@szeZdZdZdddZddZddZd d d Zd!ddZ  d"ddZ d#ddZ d$ddZ ddZ ddZ ddZdS)%ReplicationServerz Represents a server that publishes replication data. Replication change files allow to keep local OSM data up-to-date without downloading the full dataset again. osc.gzcCs||_||_t|_dS)N)baseurl diff_typerequestsSessionsession)selfurlrr;/usr/lib/python3/dist-packages/osmium/replication/server.py__init__szReplicationServer.__init__cCsddtji}tj||dS)Nz User-Agentz pyosmium/{})headers)formatrpyosmium_release urlrequestRequest)rrrrrr make_request szReplicationServer.make_requestcCs:t}|D] }|d||d<q|jj||ddS)a Download a resource from the given URL and return a byte sequence of the content. This method has no support for cookies or any special authentication methods. If you need these, you have to provide your own custom URL opener. The method has to return an object which supports the `read()` and `readline()` methods to access the content. Example:: def my_open_url(self, url): opener = urlrequest.build_opener() opener.addheaders = [('X-Fancy-Header', 'important_content')] return opener.open(url) svr = ReplicationServer() svr.open_url = my_open_url rT)rstream)dict header_itemsrget get_full_url)rrrhrrropen_url$s zReplicationServer.open_urlcCs|d}|}|}|dus||jkrdSt}|dkr`||jkr`z||}Wn tdd}Yt|dkrA||kr@dSn||||j8}td||d|d7}|dkr`||jks!t |d||jS)a Create a MergeInputReader and download diffs starting with sequence id `start_id` into it. `max_size` restricts the number of diffs that are downloaded. The download stops as soon as either a diff cannot be downloaded or the unpacked data in memory exceeds `max_size` kB. If some data was downloaded, returns a namedtuple with three fields: `id` contains the sequence id of the last downloaded diff, `reader` contains the MergeInputReader with the data and `newest` is a sequence id of the most recent diff available. Returns None if there was an error during download or no new data was available. r)Nrz(Error during diff download. Bailing out.z:Downloaded change %d. (%d kB available in download buffer)r!) get_state_infor rget_diff_blockLOGdebuglen add_bufferrr )rstart_idmax_size left_size current_idrrddiffdatarrr collect_diffs:s.  zReplicationServer.collect_diffsr*TcCs0|||}|dur dS|jj|||d|jS)a Download diffs starting with sequence id `start_id`, merge them together and then apply them to handler `handler`. `max_size` restricts the number of diffs that are downloaded. The download stops as soon as either a diff cannot be downloaded or the unpacked data in memory exceeds `max_size` kB. If `idx` is set, a location cache will be created and applied to the way nodes. You should be aware that diff files usually do not contain the complete set of nodes when a way is modified. That means that you cannot just create a new location cache, apply it to a diff and expect to get complete way geometries back. Instead you need to do an initial data import using a persistent location cache to obtain a full set of node locations and then reuse this location cache here when applying diffs. Diffs may contain multiple versions of the same object when it was changed multiple times during the period covered by the diff. If `simplify` is set to False then all versions are returned. If it is True (the default) then only the most recent version will be sent to the handler. The function returns the sequence id of the last diff that was downloaded or None if the download failed completely. N)idxsimplify)r7r applyr )rhandlerr1r2r8r9diffsrrr apply_diffsgs zReplicationServer.apply_diffsNcCs|||}|dur dSt|} | j} t} | | _|rC| d|j| dt|j | |j } | durC| d| j d|durV| D] \} }| | |qK|dur`t|}nt||}| |_t|| }td|j| || | ||j |jfS)a Download diffs starting with sequence id `start_id`, merge them with the data from the OSM file named `infile` and write the result into a file with the name `outfile`. The output file must not yet exist. `max_size` restricts the number of diffs that are downloaded. The download stops as soon as either a diff cannot be downloaded or the unpacked data in memory exceeds `max_size` kB. If `set_replication_header` is true then the URL of the replication server and the sequence id and timestamp of the last diff applied will be written into the `writer`. Note that this currently works only for the PBF format. `extra_headers` is a dict with additional header fields to be set. Most notably, the 'generator' can be set this way. `outformat` sets the format of the output file. If None, the format is determined from the file name. The function returns a tuple of last downloaded sequence id and newest available sequence id if new data has been written or None if no data was available or the download failed completely. Nosmosis_replication_base_url#osmosis_replication_sequence_numberosmosis_replication_timestampz%Y-%m-%dT%H:%M:%SZzMerging changes into OSM file.)r7oioReaderheaderhas_multiple_object_versionsHeadersetrstrr r+r strftimeitemsFileWriterr-r.r apply_to_readercloser)rinfileoutfiler1r2set_replication_header extra_headers outformatr<r has_historyr'infokvofwriterrrrapply_diffs_to_files4         z%ReplicationServer.apply_diffs_to_fileFc Cs|}|dur dS||jks|jdkr|jSd}d}|durctd|||}|durI|j|krI|jdks@|jd|jkrC|jS|}d}d}|dur_t||jd}||kr]|jS|}|dus |rqt|j|jd}n*|j|j}|j|j} ||j} |jt| | |}||jkr|jd}||} | dur|d} | dur| |jkr|| } | d8} | dur| |jks| dur|d} | dur| |jkr|| } | d7} | dur| |jks| dur|jS| j|kr| }n| }|jd|jkr|jSqd)a Get the sequence number of the replication file that contains the given timestamp. The search algorithm is optimised for replication servers that publish updates in regular intervals. For servers with irregular change file publication dates 'balanced_search` should be set to true so that a standard binary search for the sequence will be used. The default is good for all known OSM replication services. NrzTrying with Id %sr!)r+r r r-r.int total_secondsr) rr balanced_searchupperlowerloweridnewid base_splitidts_intseq_intgoalsplitsplitidrrrtimestamp_to_sequencesh          z'ReplicationServer.timestamp_to_sequencerZc Cslt|dD]}z ||||}Wnty2}ztd|t|WYd}~dSd}~wwd}d}t|dr@|j }n|j }|D]\}| d}d|vr[|d| d}n| }|r|dd } t| d krqdS| dd kr~t| d}qF| dd krz tj| dd }Wn tyYn w|jtjjd }qF|dur|durt||dSqdS)af Downloads and returns the state information for the given sequence. If the download is successful, a namedtuple with `sequence` and `timestamp` is returned, otherwise the function returns `None`. `retries` sets the number of times the download is retried when pyosmium detects a truncated state file. r!z%Loading state info %s failed with: %sN iter_lineszutf-8#r=rZsequenceNumberr z%Y-%m-%dT%H\:%M\:%SZ)tzinfo)r r )ranger(r get_state_url Exceptionr-r.rGhasattrrireadlinedecodeindexstriprfr/r[dtdatetimestrptime ValueErrorreplacetimezoneutcr) rseqretries_responseerrtsget_linelinekvrrrr+sH        z ReplicationServer.get_state_infocCs.||||}t|dr|jS|S)z Downloads the diff with the given sequence number and returns it as a byte sequence. Throws a :code:`urllib.error.HTTPError` (or :code:`urllib2.HTTPError` in python2) if the file cannot be downloaded. content)r(r get_diff_urlrqrread)rr}resprrrr,Ks z ReplicationServer.get_diff_blockcCs4|dur |jdSd|j|d|dd|dfS)z Returns the URL of the state.txt files for a given sequence id. If seq is `None` the URL for the latest state info is returned, i.e. the state file in the root directory of the replication service. Nz /state.txtz%s/%03i/%03i/%03i.state.txt@B)rrr}rrrroZs  zReplicationServer.get_state_urlcCs&d|j|d|dd|d|jfS)zE Returns the URL to the diff file for the given sequence id. z%s/%03i/%03i/%03i.%srr)rrrrrrrhs zReplicationServer.get_diff_url)r)r))r)r*T)r)TNN)F)NrZ)__name__ __module__ __qualname____doc__rr r(r7r=rYrhr+r,rorrrrrrs    -" A R/ r)rrurllib.requestrequestrrwrv collectionsrmathrosmiumrrrArlogging getLoggerr- addHandler NullHandlerrr rrrrrs