__/ [ canadafred ] on Friday 17 March 2006 02:31 \__
>> I am wondering how it is possible to place the results from a search query
>> url with wget, curl, or bash scripts (or any other command line tool) and
>> get the results(including the results of the next page) into a txt file
>> with
>> each a newline for each url of the result?
>> i.e for a search query; something like www.example.com=query?...
>> each result can be placed on a newline in a text file.
>
> cross posting removed
Thanks, Fred.
At risk of answering an Internet troll:
Use wget, e.g.:
,----[ Commands ]
| cd ~; wget http://www.google.co.uk/search?&ie=utf-8&oe=utf-8&q=nospam
`----
Then, you could use Perl (script from Brian Wakem):
,----[ get_urls.sh ]
| cat ~/index.html| perl -ne '@url=m!(http://[^>"]+)!g;print "$_\n"
| foreach @url' > ~/googleurls
`----
For more information:
,----[ Commands ]
| man wget
`----
Notice your ability to spoof user-agent, which might be helpful.
Run the scripts and wget with caution. Retain good 'Net citizenship'.
Hope it helps,
Roy
--
Roy S. Schestowitz | chmod a-r *.mbox
http://Schestowitz.com | SuSE Linux ¦ PGP-Key: 0x74572E8E
10:55am up 9 days 3:32, 7 users, load average: 0.25, 0.66, 0.78
http://iuron.com - next generation of search paradigms
|
|
