python2.7 + scrapy安装配置

为了使用scrapy这个框架,就必须安装使用python2.7 .

python2.7 的安装

yum install libbz2 readline-devel sqlite-devel zlib-devel openssl-deve
cd /usr/tmp
yum install gcc

wget https://www.python.org/ftp/python/2.7/Python-2.7.tgz
tar -xvf Python-2.7.tgz
cd Python-2.7
./configure --prefix=/var/python2.7
make && make install

mv /usr/bin/python /usr/bin/python2.6
ln -s /var/python2.7/bin/python2.7 /usr/bin/python

变更了python会导致yum无法正常使用,需修改

vi /usr/bin/yum
#把第一行
#"#!/usr/bin/python" 改为 "#!/usr/bin/python2.6"即可

安装pip (需要先安装setuptools)

wget https://pypi.python.org/packages/source/p/pip/pip-1.5.4.tar.gz
tar -xzvf pip-1.5.4.tar.gz
cd pip-1.5.4
python setup.py install

安装完成后可用
/var/python2.7/bin/pip

建立软链接
ln -s /var/python2.7/bin/pip /usr/bin/pip
就能直接用pip了

爬虫,需要安装以下(可用 yum install libxml2 libxml2-devel 安装,但试用pip操作)
/var/python2.7/bin/pip install requests
/var/python2.7/bin/pip install lxml
一大堆报错
src/lxml/lxml.etree.c:119908: error: ‘struct __pyx_obj_4lxml_5etree__SaxParserContext’ has no member named ‘_node_stack

处理

yum install libxslt-devel

再安装
/var/python2.7/bin/pip install lxml
报错
Could not find .egg-info directory in install record for lxml
处理 升级setuptools

wget https://bootstrap.pypa.io/ez_setup.py
chmod +x ./ez_setup.py
./ez_setup.py

再安装 /var/python2.7/bin/pip install lxml

出现
Requirement already satisfied (use --upgrade to upgrade)
说明已经安装成功

安装scrapy 爬虫框架
pip install scrapy

报错
Could not find .egg-info directory in install record for PyDispatcher>=2.0.5 (from Scrapy)
处理

Could not find .egg-info directory in install record for Twisted>=13.1.0 (from Scrapy)

处理
pip install twisted

>>报错
>>Could not find .egg-info directory in install record for zope.interface>=3.6.0 (from twisted)
>>处理
>>wget https://pypi.python.org/packages/source/z/zope.interface/zope.interface-4.0.5.zip
>>unzip zope.interface-4.0.5.zip
>>cd zope.interface-4.0.5
>>python setup.py install

再安装 pip install twisted

报错No package 'libffi' found

wget ftp://sourceware.org/pub/libffi/libffi-3.2.1.tar.gz
tar -zxvf libffi-3.2.1.tar.gz
cd libffi-3.2.1
./configure
make
make install

export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig:$PKG_CONFIG_PATH

重新安装scrapy
卸载pip uninstall scrapy
安装pip install scrapy

安装成功 再也没有报错了

番外:
python2.7 安装bs4
pip install beautifulsoup4

安装mysql扩展
pip install MySQL-python


标签: python, scrapy, 爬虫

非特殊说明,本博所有文章均为博主原创。

最新文章

发表评论