o .&a>@sddlZddlZddlZddlmZddlmZddlmZddl m Z m Z ddl m Z m Z mZmZddlmZddlmZeZd d Zd d ZGd ddeZGdddeZGdddeZdS)N)parse)tzlocal) ClientError)find_bucket_key get_file_stat) BucketListercreate_warningfind_dest_path_comp_key EPOCH_TIME)six)queuecCsHt|j}t|r dSt|rdSt|rdSt|r"dSdS)z This function checks to see if a special file. It checks if the file is a character special device, block special device, FIFO, or socket. TF)osstatst_modeS_ISCHRS_ISBLKS_ISFIFOS_ISSOCK)pathmoderH/usr/lib/python3/dist-packages/awscli/customizations/s3/filegenerator.pyis_special_files     rc Cstj|rzt|WdSttfyYdSwzt|d }WdWdS1s.wYWdSttfyAYdSw)z This function checks to see if a file or a directory can be read. This is tested by performing an operation that requires read access on the file or the directory. FrNT)r risdirlistdirOSErrorIOError_open)rfdrrr is_readable4s"     r cs,eZdZdZdeZfddZZS)FileDecodingErrorz1Raised when there was an issue decoding the file.zPlease check your locale settings. The filename was decoded as: %s On posix platforms, check the LC_CTYPE environment variable.cs<||_||_dt|j|j|jf|_tt||jdS)NzJThere was an error trying to decode the the file %s in directory "%s". %s) directory file_namereprADVICE error_messagesuperr!__init__)selfr"filename __class__rrr(SszFileDecodingError.__init__) __name__ __module__ __qualname____doc__sysgetfilesystemencodingr%r( __classcell__rrr+rr!Jsr!c@seZdZ   dddZdS)FileStatNc Cs:||_||_||_||_||_||_||_||_| |_dSN) srcdest compare_keysize last_updatesrc_type dest_typeoperation_name response_data) r)r6r7r8r9r:r;r<r=r>rrrr(`s zFileStat.__init__)NNNNNNNN)r-r.r/r(rrrrr4_s r4c@sveZdZdZ  dddZddZdd Zd d Zd d ZddZ ddZ ddZ ddZ ddZ ddZddZdS) FileGeneratora} This is a class the creates a generator to yield files based on information returned from the ``FileFormat`` class. It is universal in the sense that it will handle s3 files, local files, local directories, and s3 objects under the same common prefix. The generator yields corresponding ``FileInfo`` objects to send to a ``Comparator`` or ``S3Handler``. TNcCsH||_||_||_||_||_|st|_i|_|dur"||_dSdSr5)_clientr=follow_symlinks page_size result_queuer Queuerequest_parameters)r)clientr=rArBrCrErrrr(vs  zFileGenerator.__init__c cs|j|jd}|dd}|dd}|dd}||||d}|D]#\}}t||\} } || | |||jd} || |td i| Vq%dS) z This is the generalized function to yield the ``FileInfo`` objects. ``dir_op`` and ``use_src_name`` flags affect which files are used and ensure the proper destination paths and compare keys are formed. )s3localr6rtyper7dir_op)r6r7r8r;r<r=Nr) list_objects list_filesr r=_inject_extra_informationr4) r)filesfunction_tablesourcer;r< file_iteratorsrc_pathextra_information dest_pathr8file_stat_kwargsrrrcalls     zFileGenerator.callcCs8|d}|d|d<|d|d<|dkr||d<dSdS)Nr;Sizer9 LastModifiedr:rGr>r)r)rUrSr;rrrrMs    z'FileGenerator._inject_extra_informationccstjjtjjtjj}}}tjtj}}||s}|s+||}|r)|VdSdS||} g} | D]} | || sO||| } || rJ| tjj } | | q3| | tj d| D]$} ||| } || rr| | |D]} | VqkqZ|| }|r||VqZdSdS)a{ This function yields the appropriate local file or local files under a directory depending on if the operation is on a directory. For directories a depth first search is implemented in order to follow the same sorted pattern as a s3 list objects operation outputs. It yields the file's source path, size, and last update /N)r rjoinrisfileerrorrshould_ignore_file_safely_get_file_stats)should_ignore_file_with_decoding_warningssepappendnormalize_sortrL)r)rrJrZrr[r\rstats listdir_namesnamesname file_pathxrrrrLsB          zFileGenerator.list_filesc CsNzt|\}}Wnttfy||YdSw|||}|||dfS)NrWrX)rr ValueErrortriggers_warning_validate_update_time)r)rgr9r:rrrr^s z$FileGenerator._safely_get_file_statscCs*|durt|ddd}|j|tS|S)Nz?File has an invalid timestamp. Passing epoch time as timestamp.F)rr& skip_file)rrCputr )r) update_timerwarningrrrrls z#FileGenerator._validate_update_timecs|jfddddS)a The purpose of this function is to ensure that the same path seperator is used when sorting. In windows, the path operator is a backslash as opposed to a forward slash which can lead to differences in sorting between s3 and a windows machine. cs |Sr5)replace)item characteros_seprrs z.FileGenerator.normalize_sort..)keyN)sort)r)rerurtrrsrrbszFileGenerator.normalize_sortcCsNt|tjst||}tt||j}|j|dSt j ||}| |S)a We can get a UnicodeDecodeError if we try to listdir() and can't decode the contents with sys.getfilesystemencoding(). In this case listdir() returns the bytestring, which means that join(, ) could raise a UnicodeDecodeError. When this happens we warn using a FileDecodingError that provides more information into what's going on. T) isinstancer text_typer!rr$r&rCrnr rrZr])r)dirnamer*decoding_errorrprrrrr_s   z7FileGenerator.should_ignore_file_with_decoding_warningscCsP|jstj|r|tjr|dd}tj|rdS||}|r&dSdS)z This function checks whether a file should be ignored in the file generation process. This includes symlinks that are not to be followed and files that generate warnings. NTF)rAr rrendswithr`islinkrk)r)rwarning_triggeredrrrr]s   z FileGenerator.should_ignore_filecCsntj|st|d}|j|dSt|r$t|d}|j|dSt|s5t|d}|j|dSdS)a This function checks the specific types and properties of a file. If the file would cause trouble, the function adds a warning to the result queue to be printed out and returns a boolean value notify whether the file caused a warning to be generated. Files that generate warnings are skipped. Currently, this function checks for files that do not exist and files that the user does not have read access. zFile does not exist.TzHFile is character special device, block special device, FIFO, or socket.zFile/Directory is not readable.F)r rexistsrrCrnrr )r)rrprrrrks     zFileGenerator.triggers_warningc cst|\}}|s|r||VdSt|j}|jdi}|j|||j|dD](}|\}} | ddkrE|drE|j dkrD|| fVq)|sL||krLq)|| fVq)dS)z This function yields the appropriate object or objects under a common prefix depending if the operation is on objects under a common prefix. It yields the file's source path, size, and last update. ListObjectsV2)bucketprefixrB extra_argsrWrrYdeleteN) r_list_single_objectrr@rEgetrKrBr~r=) r)s3_pathrJrrlisterrrw source_pathr>rrrrK2s&       zFileGenerator.list_objectsc Cs|jdkr |dddfSt|\}}z||d}||jdi|jjd i|}Wn(tyT}z|jdddks>|j }d||dd <t|dd}~wwt | d |d <t |d }| t|d <||fS)Nrri)BucketKey HeadObjectErrorCode404zKey "%s" does not existMessage ContentLengthrWrXr)r=rupdaterErr@ head_objectrresponsecopyintpopr astimezoner)r)rrrwparamsrer:rrrrTs&      z!FileGenerator._list_single_object)TNNN)r-r.r/r0r(rVrMrLr^rlrbr_r]rkrKrrrrrr?ns    4    "r?)r r1rdateutil.parserr dateutil.tzrbotocore.exceptionsrawscli.customizations.s3.utilsrrrrr r awscli.compatr r openrrr Exceptionr!objectr4r?rrrrs