07-Requests 库的基本使用

发布时间 2020年5月2日 • 5 分钟读完 • 854 字

分享链接

Link copied to clipboard

Python 的 Requests 库的基本使用

Requests

利用 Python 现有的库可以非常方便地实现网络请求的模拟，常见的库有 urllib、requests等

安装 Requests

requests 库是 Python 的一个第三方库，需要额外安装，在这之前需要先按照好 Python3 环境，

接着使用 pip3 即可轻松地安装好 requests 库：pip3 install requests

实例引入

用 Python 写爬虫的第一步就是模拟发起一个请求，把网页的源代码获取下来

示例网站：

基础用法

Get 请求

如果客户端发起的是 GET 请求的话，该网站会判断并返回相应的请求信息，包括 Headers、IP等

r = requests.get('http://httpbin.org/get')
print(r.text)

运行结果

{
  "args": {}, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate", 
    "Host": "httpbin.org", 
    "User-Agent": "python-requests/2.23.0", 
    "X-Amzn-Trace-Id": "Root=1-5ead7abd-1514fc83002543dc47c3a5dd"
  }, 
  "origin": "124.77.82.90", 
  "url": "http://httpbin.org/get"
}

如果想添加两个参数，其中 name 是 germey，age 是 25

r = requests.get('http://httpbin.org/get?name=germey&age=25')
print(r.text)

如果响应结果是一个 JSON 格式数据，可以直接调用 json 方法，解析结果

r = requests.get('http://httpbin.org/get?name=germey&age=25')
print(r.text)

data = {
    'name': 'germey',
    'age': 25
}
r = requests.get('http://httpbin.org/get', params=data)

print(type(r.text))
print(r.json())
print(type(r.json()))

运行结果

<class 'str'>
{'args': {'age': '25', 'name': 'germey'}, 'headers': {'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'Host': 'httpbin.org', 'User-Agent': 'python-requests/2.23.0', 'X-Amzn-Trace-Id': 'Root=1-5eada1af-098f178c912c6d648ae17708'}, 'origin': '124.77.82.90', 'url': 'http://httpbin.org/get?name=germey&age=25'}
<class 'dict'>

抓取网页

import requests
import re

r = requests.get('http://static1.scrape.cuiqingcai.com/', verify=False)
pattern = re.compile('<h2.*?>(.*?)</h2>', re.S)
titles = re.findall(pattern, r.text)
print(titles)

运行结果

['霸王别姬 - Farewell My Concubine', '这个杀手不太冷 - Léon', '肖申克的救赎 - The Shawshank Redemption', '泰坦尼克号 - Titanic', '罗马假日 - Roman Holiday', '唐伯虎点秋香 - Flirting Scholar', '乱世佳人 - Gone with the Wind', '喜剧之王 - The King of Comedy', '楚门的世界 - The Truman Show', '狮子王 - The Lion King']

抓取二进制数据

图片、音频、视频这些文件本质上都是由二进制码组成，想要抓取他们，就要拿到它们的二进制数据

r = requests.get('https://github.com/favicon.ico')
print(r.text)
print(r.content)

运行结果

:�������OL������������������!������4@���8���....����t7�������������������������>������>( @ +

b'\x00\x00\x01\x00\x02\x00\x10\x10\x00\x00\x01\x00 \x00(\x05\x00\x00&\x00\x00\x00  \x00\x00\x01\x00 \x00(\x14\x00\x00N\x05\x00\x00(\x00\x00\x00\x10\x00\x00\x00 \x00\x00\x00\x01\x00......

注意：前者出现乱码是由于图片是二进制数据，在打印时转化为 str 类型，后者是带有一个 b 这代表是 bytes 类型

添加 headers

import requests
headers = {
    'User-Agent': 'Mozilla/5.0 (iPhone; CPU iPhone OS 13_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148 MicroMessenger/7.0.12(0x17000c2d) NetType/WIFI Language/zh_CN'
}
r = requests.get('https://github.com', headers=headers)
print(r.text)

Post 请求

import requests

data = {'name': 'germey', 'age': '25'}
r = requests.post("http://httpbin.org/post", data=data)

print(r.text)

# 响应
print(type(r.status_code), r.status_code)
print(type(r.headers), r.headers)
print(type(r.cookies), r.cookies)
print(type(r.url), r.url)
print(type(r.history), r.history)

运行结果

<class 'int'> 200
<class 'requests.structures.CaseInsensitiveDict'> {'Date': 'Sat, 02 May 2020 16:47:57 GMT', 'Content-Type': 'application/json', 'Content-Length': '498', 'Connection': 'keep-alive', 'Server': 'gunicorn/19.9.0', 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Credentials': 'true'}
<class 'requests.cookies.RequestsCookieJar'> <RequestsCookieJar[]>
<class 'str'> http://httpbin.org/post
<class 'list'> []

requests 还提供了一个内置的状态码查询对象 requests.codes

r = requests.post("http://httpbin.org/post")
exit() if not r.status_code == requests.codes.ok else print('Rquest successfully')

高级用法

文件上传

import requests
files = {'file': open('favicon.ico', 'rb')}
r = requests.post('http://httpbin.org/post', files=files)
print(r.text)

运行结果

{
  "args": {}, 
  "data": "", 
  "files": {
    "file": "data:application/octet-stream;base64,AAABAAIAEBAAAAEAIAAoBQAAJgAAACAgAAABACAAKBQAAE4FAAAoAAAAEAAAACAAAAABACAAAAAAAA........"
  }, 
  "form": {}, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate", 
    "Content-Length": "6665", 
    "Content-Type": "multipart/form-data; boundary=042924f0ed0d7c7dcc291242ae86b9e5", 
    "Host": "httpbin.org", 
    "User-Agent": "python-requests/2.23.0", 
    "X-Amzn-Trace-Id": "Root=1-5eada504-8a914287cf91dc1d48a771ea"
  }, 
  "json": null, 
  "origin": "124.77.82.90", 
  "url": "http://httpbin.org/post"
}

Cookies

import requests

r = requests.get('http://www.baidu.com')
print(r.cookies)
for key, value in r.cookies.items():
    print(key + '=' + value)

运行结果

<RequestsCookieJar[<Cookie BDORZ=27315 for .baidu.com/>]>
BDORZ=27315

使用 Cookie 这里以 github.com 网站 cookie 为例

headers = {
    'Cookie': '_octo=GH1.1.1686548572.1563008144; _ga=GA1.2.1911027902.1563008150; tz=Asia%2FShanghai; _device_id=86bff5bb8b383bf5b012cdb88eb59d10; ignored_unsupported_browser_notice=false; logged_in=yes; dotcom_user=dbing; user_session=nlCeShp0HeBErMxoREDDUcJZ8oq6zzXx9ts3QnyyxFcpNC8t; __Host-user_session_same_site=nlCeShp0HeBErMxoREDDUcJZ8oq6zzXx9ts3QnyyxFcpNC8t; tz=Asia%2FShanghai; has_recent_activity=1; ......',
    'User-Agent': 'Mozilla/5.0 (iPhone; CPU iPhone OS 13_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148 MicroMessenger/7.0.12(0x17000c2d) NetType/WIFI Language/zh_CN'
}

r = requests.get('https://github.com/', headers=headers)
print(r.text)

构造 RequestsCookieJar 对象，将刚才复制的 Cookie 处理下并赋值

import requests

headers = {
    'Cookie': '_octo=GH1.1.1686548572.1563008144; _ga=GA1.2.1911027902.1563008150; tz=Asia%2FShanghai; _device_id=86bff5bb8b383bf5b012cdb88eb59d10; ignored_unsupported_browser_notice=false; logged_in=yes; dotcom_user=dbing; user_session=nlCeShp0HeBErMxoREDDUcJZ8oq6zzXx9ts3QnyyxFcpNC8t; __Host-user_session_same_site=nlCeShp0HeBErMxoREDDUcJZ8oq6zzXx9ts3QnyyxFcpNC8t; tz=Asia%2FShanghai; has_recent_activity=1; ......',
    'User-Agent': 'Mozilla/5.0 (iPhone; CPU iPhone OS 13_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148 MicroMessenger/7.0.12(0x17000c2d) NetType/WIFI Language/zh_CN'
}

for cookie in cookies.split(';'):
    key, value = cookie.split('=', 1)
    jar.set(key,value)

r = requests.get('https://github.com/', cookies=jar, headers=headers)
print(r.text)

Session 维持

requests.get('http://httpbin.org/cookies/set/number/123456789')
r = requests.get('http://httpbin.org/cookies')
print(r.text)
# 结果
{
  "cookies": {}
}

s = requests.Session()
s.get('http://httpbin.org/cookies/set/number/123456789')
r = s.get('http://httpbin.org/cookies')
print(r.text)
# 结果
{
  "cookies": {
    "number": "123456789"
  }
}

SSL 证书验证

很多网站都要求使用 HTTPS 协议，但是有些网站没有设置好 HTTPS 证书，或者网站的 HTTPS 证书不被 CA 机构认可，这些网站就会出现 SSL 证书错误的提示：您的链接不是私密链接

import requests
response = requests.get('https://static1.scrape.cuiqingcai.com/')
print(response.status_code)

运行结果

发生异常: SSLError
HTTPSConnectionPool(host='static1.scrape.cuiqingcai.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])")))

import requests
response = requests.get('https://static1.scrape.cuiqingcai.com/', verify=False)
print(response.status_code)

结果如下

InsecureRequestWarning: Unverified HTTPS request is being made to host 'static1.scrape.cuiqingcai.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning,
200

通过设置忽略警告的方式来屏蔽这个警告

import requests
from requests.packages import urllib3

urllib3.disable_warnings()
response = requests.get('https://static1.scrape.cuiqingcai.com/', verify=False)
print(response.status_code)
# 结果
200

通过捕获警告日志的方式忽略警告

import logging
import requests

logging.captureWarnings(True)
response = requests.get('https://static1.scrape.cuiqingcai.com/', verify=False)
print(response.status_code)

可以指定一个本地证书用作客户端证书

import requests

response = requests.get('https://static1.scrape.cuiqingcai.com/', cert=('/path/server.crt','/path/server.key'))
print(response.status_code)

可以是单个文件（包含秘钥和证书）或一个包含两个文件路径的元祖

超时设置

为了防止服务器不能及时响应而报错，应该设置一个超时时间，这需要用到 timeout 参数

r = requests.get('http://httpbin.org/get', timeout=1)
# r = requests.get('http://httpbin.org/get', timeout=(5,30))
# 永久等待-或者留空不设置
# r = requests.get('http://httpbin.org/get', timeout=None)
print(r.status_code)

身份认证

访问某些设置了身份认证的网站时，有时需要登录

可以使用 requests 自带的身份认证功能，通过 auth 参数即可设置

import requests
from requests.auth import HTTPBasicAuth

r = requests.get('https://static3.scrape.cuiqingcai.com/', verify=False, auth=HTTPBasicAuth('admin','admin'))
print(r.status_code)

简写

r = requests.get('https://static3.scrape.cuiqingcai.com/', verify=False, auth=('admin', 'admin'))
print(r.status_code)

OAuth 认证

requests 还提供了其他认证方式，如 OAuth 认证，不过需要安装 oauth 包，

pip3 安装命令：pip3 install requests_oauthlib conda 安装命令：conda install -c anaconda requests-oauthlib

import requests
from requests_oauthlib import OAuth1

url = 'https://api.twitter.com/1.1/account/verify_credentials.json'
auth = OAuth1('YOUR_APP_KEY', 'YOUR_APP_SECRET', 'USER_OAUTH_TOKEN', 'USER_OAUTH_TOKEN_SECRET')
requests.get(url, auth=auth)

requests-oauthlib 的官方文档：

HTTP 协议代理

某些网站测试时候请求几次，能正常获取内容，但是对于大规模频繁的请求，网站可能会弹出验证码，或者跳转到登录认证页面，还有可能会直接封禁客户端的 IP，导致一定时间段内无法访问，为了防止这种事情发生，

需要设置代理来解决这个问题，这就需要用到 proxies 参数

import requests

proxies = {
    'http': 'http://10.10.10.10:1080',
    'https': 'http://10.10.10.10:1080'
}
requests.get('https://httpbin.org/get', proxies=proxies)

proxies = {'https': 'http://user:password@10.10.10.10:1080/',}
requests.get('https://httpbin.org/get', proxies)

SOCKS 协议代理

requests 还支持 SOCKS 协议的代理

需要安装 socks 库，命令:pip3 install "requests[socks]"

import requests
proxies = {
    'http': 'socks5://user:password@host:port',
    'https': 'socks5://user:password@host:port'
}
requests.get('https://httpbin.org/get', proxies=proxies)

Prepared Request

from requests import Request, Session

url = 'http://httpbin.org/post'
data = {'name': 'germey'}
headers = {
    'User-Agent': 'Mozilla/5.0 (iPhone; CPU iPhone OS 13_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148 MicroMessenger/7.0.12(0x17000c2d) NetType/WIFI Language/zh_CN'
}
s = Session()
req = Request('POST', url, data=data, headers=headers)
prepped = s.prepare_request(req)
r = s.send(prepped)
print(r.text)

结果

{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {
    "name": "germey"
  }, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate", 
    "Content-Length": "11", 
    "Content-Type": "application/x-www-form-urlencoded", 
    "Host": "httpbin.org", 
    "User-Agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 13_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148 MicroMessenger/7.0.12(0x17000c2d) NetType/WIFI Language/zh_CN",
 
    "X-Amzn-Trace-Id": "Root=1-5ead9f0d-9853e7e448cf117256717562"
  }, 
  "json": null, 
  "origin": "124.77.82.90", 
  "url": "http://httpbin.org/post"
}

更多用法可以参考 Requests 官方文档

The End!

docker-composer 中升级PHP到7.4版本遇到的问题

05-多线程的基本原理

关注我

我的工作是编码和发布开发者表情包

Requests

安装 Requests

实例引入

基础用法

Get 请求

抓取网页

抓取二进制数据

添加 headers

Post 请求

高级用法

文件上传

Cookies

Session 维持

SSL 证书验证

超时设置

身份认证

OAuth 认证

HTTP 协议代理

SOCKS 协议代理

Prepared Request

评论