简单记录一下某个页面来路的搜索关键字，用 python 正则获取request.headers的referer字段。
referer 分析
先看看几个搜索引擎的referer：
1
2
3
4
5
6
7
8
9
10
11
12
13
http://www.baidu.com/s?q=%E5%AE%9D%E5%AE%9D%E6%88%90%E9%95%BF%E6%97%A5%E8%AE%B0&sa=%E7%99%BE%E5%BA%A6+%E6%90%9C%E7%B4%A2&prog=aff&client=&channel=&hl=zh-CN&source=sdo_sb&wd=%E5%AE%9D%E5%AE%9D%E6%88%90%E9%95%BF%E6%97%A5%E8%AE%B0
http://www.baidu.com/s?wd=%E5%AE%9D%E5%AE%9D%E6%88%90%E9%95%BF%E6%97%A5%E8%AE%B0&rsp=0&f=1&oq=%E5%AE%9D%E5%AE%9D%E6%88%90%E9%95%BF%E6%97%A5%E8%AE%B0%E5%8D%9A%E5%AE%A2&tn=baiduhome_pg&ie=utf-8&rsv_ers=xn1&rs_src=0
http://www.baidu.com/s?wd=%E5%AE%9D%E5%AE%9D%E6%88%90%E9%95%BF%E6%97%A5%E8%AE%B0&rsv_spt=1&issp=1&rsv_bp=0&ie=utf-8&tn=baiduhome_pg&n=2&inputT=1873
http://www.baidu.com/s?word=%E5%AE%9D%E5%AE%9D%E6%88%90%E9%95%BF%E6%97%A5%E8%AE%B0&tn=sitehao123&ie=utf-8
http://www.google.com.hk/search?q=%E5%AE%9D%E5%AE%9D%E6%88%90%E9%95%BF%E6%97%A5%E8%AE%B0&sa=Google+%E6%90%9C%E7%B4%A2&prog=aff&client=&channel=&hl=zh-CN&source=sdo_sb&wd=
http://www.google.com/#hl=en&newwindow=1&output=search&sclient=psy-ab&q=%E5%AE%9D%E5%AE%9D%E6%88%90%E9%95%BF%E6%97%A5%E8%AE%B0&oq=%E5%AE%9D%E5%AE%9D%E6%88%90%E9%95%BF%E6%97%A5%E8%AE%B0&gs_l=hp.12...909.909.0.2701.1.1.0.0.0.0.149.149.0j1.1.0.les%3B..0.0...1c.4IJDxm3b3Og&pbx=1&bav=on.2,or.r_gc.r_pw.r_qf.&fp=956c9e2b545c788f&biw=1280&bih=684
http://www.360sou.com/s?ie=utf-8&src=hao_phome&_re=1&q=%E5%AE%9D%E5%AE%9D%E6%88%90%E9%95%BF%E6%97%A5%E8%AE%B0
http://www.sogou.com/web?query=%B1%A6%B1%A6%B3%C9%B3%A4%C8%D5%BC%C7&_asf=www.sogou.com&_ast=1346601462&w=01019900&p=40040100&sut=2405&sst0=1346601461745
http://www.yahoo.cn/s?src=8003&vendor=100101&source=ycnhp_search_button&q=%E5%AE%9D%E5%AE%9D%E6%88%90%E9%95%BF%E6%97%A5%E8%AE%B0
http://cn.bing.com/search?q=%E5%AE%9D%E5%AE%9D%E6%88%90%E9%95%BF%E6%97%A5%E8%AE%B0%E6%9C%AC&go=&qs=n&form=QBRE&pq=%E5%AE%9D%E5%AE%9D%E6%88%90%E9%95%BF%E6%97%A5%E8%AE%B0%E6%9C%AC&sc=0-6&sp=-1&sk=
http://www.youdao.com/search?q=sae+blog
http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&ved=0CB8QFjAA&url=http%3A%2F%2Fgobaby.sinaapp.com%2F&ei=YYtFUMPJF_GNiAeN34GADQ&usg=AFQjCNEIdqv8dKxhRFo1LxYK7TjmcAHnXQ&sig2=2NcnpgwihH0gKCcs8o3RIw
总结
- 搜索引擎host：baidu|google|360sou|sogou|yahoo|bing|youdao
- url的GET字段：wd|word|q|query|pq
- 当点击百度搜索结果页面下方的“相关搜索”时会有一个“oq”字段，其值应该是old query string吧。
识别
python正则实现提取搜索关键字的代码：
1
2
3
4
5
6
7
8
9
10
11
12
import re
QR_RE = re.compile(r'(baidu|google|360sou|sogou|yahoo|bing|youdao).+?[\?\&]{1}(wd|word|q|query|pq)=([^&]+)', re.I)
def get_search_kw(referer):
if referer:
matchobj = QR_RE.search(referer)
if matchobj:
try:
return unquote(matchobj.group(3)).decode('utf-8')
except:
pass
return ''
注意，当用Chrome打开 https://www.google.com 搜索时其referer字段不包含搜索关键字。
本文网址: https://pylist.com/topic/22.html 转摘请注明来源