Re: [android-developers] Crawling the Google Play store: Research Project

I had thought that some of those APIs worked.

In particular, what steps are you stuck on in using those libraries?
If you can't use the API, writing your own crawler will probably be
somewhat difficult.

Kris

On Sun, Jun 9, 2013 at 7:28 PM, Ishan Sharma <ishansharma.in@gmail.com> wrote:
> I tried the last two links which use the crawler from the second link. I'm
> stuck at one of the steps in each of the two. I've written to the author of
> the blog and expecting a reply. Besides these blog posts I didn't find any
> other technique so far.
>
> On Mon, Jun 10, 2013 at 3:32 AM, Kristopher Micinski
> <krismicinski@gmail.com> wrote:
>>
>> And none of those APIs worked for you?
>>
>> Kris
>>
>> On Sun, Jun 9, 2013 at 1:47 PM, Ishan Sharma <ishansharma.in@gmail.com>
>> wrote:
>> > Hello!
>> >
>> > I am working on a research project in which I need the dataset of all
>> > android application webpages from the Google Play store. I want to crawl
>> > the
>> > Google Play store to download the web pages of all the android
>> > application
>> > (All the webpages with the following base url:
>> > https://play.google.com/store/apps/). I checked the robots.txt file of
>> > the
>> > play store and it disallows crawling these URLs.
>> >
>> > Also, when I browse the Google Play store I can only see top
>> > applications up
>> > to 3 pages for each of the categories. How can I get the other
>> > application
>> > pages?
>> >
>> > If anyone has tried crawling the Google Play please let me know the
>> > following things: a) Were you successful in crawling the play store. If
>> > yes,
>> > please let me know how you did that. b) How to crawl the hidden
>> > application
>> > pages not visible in top apps for each of the categories? c) Is there a
>> > techniques to download the applications also and not just the webpages?
>> >
>> > I already searched around and found the following links:
>> >
>> > a) https://code.google.com/p/android-market-api/
>> > b) https://code.google.com/p/android-marketplace-crawler/source/checkout
>> > c)
>> >
>> > http://mohsin-junaid.blogspot.co.uk/2012/12/how-to-install-android-marketplace.html
>> > d)
>> >
>> > http://mohsin-junaid.blogspot.in/2012/12/how-to-download-multiple-android-apks.html
>> >
>> > --
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups "Android Developers" group.
>> > To post to this group, send email to android-developers@googlegroups.com
>> > To unsubscribe from this group, send email to
>> > android-developers+unsubscribe@googlegroups.com
>> > For more options, visit this group at
>> > http://groups.google.com/group/android-developers?hl=en
>> > ---
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "Android Developers" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an
>> > email to android-developers+unsubscribe@googlegroups.com.
>> > For more options, visit https://groups.google.com/groups/opt_out.
>> >
>> >
>>
>> --
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Android Developers" group.
>> To post to this group, send email to android-developers@googlegroups.com
>> To unsubscribe from this group, send email to
>> android-developers+unsubscribe@googlegroups.com
>> For more options, visit this group at
>> http://groups.google.com/group/android-developers?hl=en
>> ---
>> You received this message because you are subscribed to a topic in the
>> Google Groups "Android Developers" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/android-developers/-uYCkQVrINM/unsubscribe?hl=en.
>> To unsubscribe from this group and all its topics, send an email to
>> android-developers+unsubscribe@googlegroups.com.
>>
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>>
>
> --
> --
> You received this message because you are subscribed to the Google
> Groups "Android Developers" group.
> To post to this group, send email to android-developers@googlegroups.com
> To unsubscribe from this group, send email to
> android-developers+unsubscribe@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/android-developers?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Android Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to android-developers+unsubscribe@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

--
--
You received this message because you are subscribed to the Google
Groups "Android Developers" group.
To post to this group, send email to android-developers@googlegroups.com
To unsubscribe from this group, send email to
android-developers+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/android-developers?hl=en
---
You received this message because you are subscribed to the Google Groups "Android Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to android-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.