Spencer/Python_HTTP_JavaScript_Scraper
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

small doc changes

Browse Source
This commit is contained in:
Pin
2022-01-23 17:09:00 -05:00
parent f1e76d8d41
commit 47a4850f00
2 changed files with 12 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files

13
README.md
View File

@@ -3,11 +3,18 @@
Run the code with the following: Run the code with the following:
```bash ```bash
# python3 main.py {proto}://{site}
python3 main.py https://www.nintendo.com python3 main.py https://www.nintendo.com
``` ```
Any site that does not serve HTTP/2 content will return a 301 HTTP response code and fail out of the program. Any site that only serves HTTP/2 content will return a 301 HTTP response code.
This can be seen by plugging in `https://www.rit.edu` into the input.
It will return a 301 trying to redirect you to HTTP/2 on the same address and port. It will return a 301 trying to redirect you to HTTP/2 on the same address and port.
No additional dependencies are required.
Any status code other than 200 will be reported back to the user without any additional processing.
The output of the program will contain the unique number of external resources.
If two files are referenced from `abc.com` they will only count as one external reference.
Any reference that shares the same base URL will count once.
A full list of references will also be printed out regardless of the uniqueness.
No additional dependencies are required to run the program.

4
main.py
View File

@@ -18,7 +18,7 @@ def generate_request():
global USER_AGENT global USER_AGENT
global REQUEST global REQUEST
USER_AGENT = "Mozilla/5.0 (X11; Linux x86_64; rv:95.0) Gecko/20100101 Firefox/95.0" USER_AGENT = "Mozilla/5.0 (X11; Linux x86_64; rv:95.0) Gecko/20100101 Firefox/95.0"
REQUEST = "GET / HTTP/1.1\r\nHost: %s\r\nConnection: close\r\nUser-Agent: %s\r\n\r\n" % (HOST_REQUEST, USER_AGENT) REQUEST = "GET / HTTP/1.1\r\nHost: %s\r\nConnection: close\r\nAccept-Encoding: Identity\r\nUser-Agent: %s\r\n\r\n" % (HOST_REQUEST, USER_AGENT)
return return
def pull_external_url(text): def pull_external_url(text):
@@ -187,7 +187,7 @@ def main():
# Obtain data from response # Obtain data from response
fr = "" fr = ""
while True: while True:
r = w.recv(8192) r = w.recv(32768)
if not r: if not r:
break break
fr += r.decode() fr += r.decode()