⚝
One Hat Cyber Team
⚝
Your IP:
216.73.216.30
Server IP:
45.79.8.107
Server:
Linux localhost 5.15.0-140-generic #150-Ubuntu SMP Sat Apr 12 06:00:09 UTC 2025 x86_64
Server Software:
nginx/1.18.0
PHP Version:
8.1.2-1ubuntu2.21
Buat File
|
Buat Folder
Eksekusi
Dir :
~
/
lib
/
python3.10
/
urllib
/
__pycache__
/
View File Name :
robotparser.cpython-310.pyc
o }ò5hÐ$ ã @ s\ d Z ddlZddlZddlZdgZe dd¡ZG dd„ dƒZG dd„ dƒZ G d d „ d ƒZ dS ) a% robotparser.py Copyright (C) 2000 Bastian Kleineidam You can choose between two licenses when using this package: 1) GNU GPLv2 2) PSF license for Python 2.2 The robots.txt Exclusion Protocol is implemented as specified in http://www.robotstxt.org/norobots-rfc.txt é NÚRobotFileParserÚ RequestRatezrequests secondsc @ sr e Zd ZdZd dd„Zdd„ Zdd„ Zd d „ Zd d „ Zd d„ Z dd„ Z dd„ Z dd„ Z dd„ Z dd„ Zdd„ ZdS ) r zs This class provides a set of methods to read, parse and answer questions about a single robots.txt file. Ú c C s2 g | _ g | _d | _d| _d| _| |¡ d| _d S )NFr )ÚentriesÚsitemapsÚ default_entryÚ disallow_allÚ allow_allÚset_urlÚ last_checked©ÚselfÚurl© r ú)/usr/lib/python3.10/urllib/robotparser.pyÚ__init__ s zRobotFileParser.__init__c C s | j S )z·Returns the time the robots.txt file was last fetched. This is useful for long-running web spiders that need to check for new robots.txt files periodically. )r ©r r r r Úmtime% s zRobotFileParser.mtimec C s ddl }| ¡ | _dS )zYSets the time the robots.txt file was last fetched to the current time. r N)Útimer )r r r r r Úmodified. s zRobotFileParser.modifiedc C s& || _ tj |¡dd… \| _| _dS )z,Sets the URL referring to a robots.txt file.é é N)r ÚurllibÚparseÚurlparseÚhostÚpathr r r r r 6 s zRobotFileParser.set_urlc C s´ z t j | j¡}W n@ t jjyI } z2|jdv r d| _n|jdkr0|jdk r>d| _W Y d}~dS W Y d}~dS W Y d}~dS W Y d}~dS d}~ww | ¡ }| | d¡ ¡ ¡ dS )z4Reads the robots.txt URL and feeds it to the parser.)i‘ i“ Ti iô Nzutf-8) r ÚrequestÚurlopenr ÚerrorÚ HTTPErrorÚcoder r Úreadr ÚdecodeÚ splitlines)r ÚfÚerrÚrawr r r r" ; s ÿÿ€ýzRobotFileParser.readc C s2 d|j v r| jd u r|| _d S d S | j |¡ d S ©NÚ*)Ú useragentsr r Úappend)r Úentryr r r Ú _add_entryH s þzRobotFileParser._add_entryc C sJ d}t ƒ }| ¡ |D ] }|s(|dkrt ƒ }d}n|dkr(| |¡ t ƒ }d}| d¡}|dkr7|d|… }| ¡ }|s>q | dd¡}t|ƒdkr|d ¡ ¡ |d< tj |d ¡ ¡|d< |d dkr~|dkrs| |¡ t ƒ }|j |d ¡ d}q |d dkr–|dkr•|j t|d d ƒ¡ d}q |d d kr®|dkr|j t|d d ƒ¡ d}q |d d krÊ|dkrÉ|d ¡ ¡ rÇt|d ƒ|_d}q |d d kr|dkr|d d¡}t|ƒdkr|d ¡ ¡ r|d ¡ ¡ rtt|d ƒt|d ƒƒ|_d}q |d dkr| j |d ¡ q |dkr#| |¡ dS dS )z”Parse the input lines from a robots.txt file. We allow that a user-agent: line is not preceded by one or more blank lines. r r é ú#Nú:z user-agentÚdisallowFÚallowTz crawl-delayz request-rateú/Úsitemap)ÚEntryr r- ÚfindÚstripÚsplitÚlenÚlowerr r Úunquoter* r+ Ú rulelinesÚRuleLineÚisdigitÚintÚdelayr Úreq_rater )r ÚlinesÚstater, ÚlineÚiÚnumbersr r r r Q sv € € € ÿ €€ ÿzRobotFileParser.parsec C s | j rdS | jr dS | jsdS tj tj |¡¡}tj dd|j|j |j |j f¡}tj |¡}|s3d}| j D ]}| |¡rD| |¡ S q6| jrN| j |¡S dS )z=using the parsed robots.txt decide if useragent can fetch urlFTr r3 )r r r r r r r; Ú urlunparser ÚparamsÚqueryÚfragmentÚquoter Ú applies_toÚ allowancer )r Ú useragentr Ú parsed_urlr, r r r Ú can_fetchš s( ÿ ÿ zRobotFileParser.can_fetchc C ó> | ¡ sd S | jD ] }| |¡r|j S q | jr | jjS d S ©N)r r rL r@ r ©r rN r, r r r Ú crawl_delay· ó ÿzRobotFileParser.crawl_delayc C rQ rR )r r rL rA r rS r r r Ú request_rateÁ rU z RobotFileParser.request_ratec C s | j sd S | j S rR )r r r r r Ú site_mapsË s zRobotFileParser.site_mapsc C s, | j }| jd ur|| jg }d tt|ƒ¡S )Nz )r r ÚjoinÚmapÚstr)r r r r r Ú__str__Ð s zRobotFileParser.__str__N)r )Ú__name__Ú __module__Ú __qualname__Ú__doc__r r r r r" r- r rP rT rV rW r[ r r r r r s I c @ s( e Zd ZdZdd„ Zdd„ Zdd„ ZdS ) r= zoA rule line is a single "Allow:" (allowance==True) or "Disallow:" (allowance==False) followed by a path.c C s< |dkr|sd}t j t j |¡¡}t j |¡| _|| _d S )Nr T)r r rG r rK r rM )r r rM r r r r Ú s zRuleLine.__init__c C s | j dkp | | j ¡S r( )r Ú startswith)r Úfilenamer r r rL â s zRuleLine.applies_toc C s | j rdndd | j S )NÚAllowÚDisallowz: )rM r r r r r r[ å s zRuleLine.__str__N)r\ r] r^ r_ r rL r[ r r r r r= × s r= c @ s0 e Zd ZdZdd„ Zdd„ Zdd„ Zdd „ Zd S ) r5 z?An entry has one or more user-agents and zero or more rulelinesc C s g | _ g | _d | _d | _d S rR )r* r< r@ rA r r r r r ë s zEntry.__init__c C s‚ g }| j D ] }| d|› ¡ q| jd ur | d| j› ¡ | jd ur3| j}| d|j› d|j› ¡ | tt| j ƒ¡ d |¡S )Nz User-agent: z Crawl-delay: zRequest-rate: r3 Ú ) r* r+ r@ rA ÚrequestsÚsecondsÚextendrY rZ r< rX )r ÚretÚagentÚrater r r r[ ñ s z Entry.__str__c C sF | d¡d ¡ }| jD ]}|dkr dS | ¡ }||v r dS q dS )z2check if this entry applies to the specified agentr3 r r) TF)r8 r: r* )r rN ri r r r rL ý s ÿzEntry.applies_toc C s$ | j D ] }| |¡r|j S qdS )zZPreconditions: - our agent applies to this entry - filename is URL decodedT)r< rL rM )r ra rD r r r rM s ÿzEntry.allowanceN)r\ r] r^ r_ r r[ rL rM r r r r r5 é s r5 ) r_ Ú collectionsÚ urllib.parser Úurllib.requestÚ__all__Ú namedtupler r r= r5 r r r r Ú<module> s B