While I was working on migrating some scrapy spiders from project to another one, I was getting the following error when I try to run any scrapy shell
As am working on a Scrapy project, I wanted to store all spider statistics to Database so as I can access it later, So I wrote the following extension.
I was working on migrating data from Radient 0.8.1 (Ruby on Rails CMS) to Joomla 2.5.6 (PHP CMS), and it was a bit silly but interesting task. So, I wrote the following simple php script to migrate articles, but you should adjust some variables first.
I think it should be developed later to use PHP & MySQL and store some details in database.
I was working on rails project and I faced this problem, my development environment DB is MySQL while production environment DB is PostgreSQL, and I wanted to move some data. I found the following 2 ways :
If you are using linux and wants to block all incoming requests to a specific port except a specific IP (your static IP or localhost in my example) , You should first block all incoming requests to this PORT using the following command :
Then, Allow this specific IP using the following command :
بعد حوالي سنة من الشغل مع شركة eSpace و وزارة التنمية الادارية في مواقع الاستفتاء و الانتخابات البرلمانية و اخيرا موقع الانتخابات الرئيسية حبيت اتكلم شوية عن اللي الواحد اتعلمه و شافه في الكواليس :)يمكن اي حد حيقرأ الكلام ده حيقول و ماله ديه ناس بتعمل شغلها و بتقبض علشان تعمل كده، اسمح لي اقولك لو اللي شغالين في المشروع ده مجرد موظفين كان اخركو حتشوفوا موقع معمول بـ Microsoft Word و بدل متشوف انت...
We have no dress code, Actually I spent most of last summer wearing shorts ! Flexi-Hours, join whenever you are ready to work ! Open Management Meeting, a weekly meeting that gather the whole company staring office boy to the CEO to discuss anything regarding the company !! Our office boy is rarely to find, he’s really funny when u ask him for a drink and he tells you “la2 kfaya...
Ref. to the previous post (Using Scrapy with proxies), I mentioned how to use a SINGLE proxy with Scrapy.
Now, what if you have different proxies ? here are a simple few changes to make it .
1. Add a new array with your proxies to your config file as follows :
2. Update your middlewares.py file to the following :
That’s it 🙂 !
I’m working currently on a scraping some websites for B-kam.com. I used to develop in PHP but when I searched for best scraping / crawling, I found Scrapy (written in Python) is the best. You can read more about it and how to start here : I searched a lot for how to use proxies with Scrapy but couldn’t find simple / Straight forward way to do it. All are talking about Middlewares and...