Fetching Data from any Website

This Project idea struck in my mind during the time of IPL(Indian Premier League) when we used to be in our class attending lecture and we guys are so much curious about the live status of the match. At that time I thought of a small screen on which we can get regular updates of the match. And the Research Begins………

Visualizing it in block diagram wise, first of all we need search for a website on which we can get the live updates of whatever we want like live stock prices,current time, current temperature,YouTube Views,likes and many more. Later we need to fetch the data from the server of that website and make an API(Application Program Interface). API serves the purpose of getting access to only those data which we are in need of from all the data available on the server.Suppose we need only number of views of our video uploaded on YouTube. So we need a interface which rather than requesting the whole HTML page, requests only number of Views of our video. Than we will give a GET request as we want to fetch the data from the server. All this task will be done throughThingspeak.com .A great website for your IoT based projects.



In thingspeak.com, we will get our required API. To test the API just request the link provided by thingspeak(On the right side of the page) in your browser than you will only get the data for what you requested.

 

In this Image, I have created an API of number of views of my YouTube Video. Method I selected is GET as we want to fetch the data and HTTP version is 1.0. Than I Inserted URL of the page and for the last row (Parse String) you need to follow following steps.

 

Now paste that XPath into parse string row and thats it save it and API will automatically get generated. Test it by pasting that link into your browser. Than it will show something like this.
This is all about to pick up the right data.But what if we want to make project based on this. To make this data more useful you will require Arduino, LCD and the most amazing thing ESP8266 ( WiFi Module).

ESP8266(WIFI MODULE)

This is an absolutely amazing thing which will help your controller to communicate to the virtual world.And it is very easy to use. It has a on chip controller and we can communicate with this WiFi Module via AT Commands.Here is the list of all AT Commands.
If you are using Arduino Mega than you can use Serial1(RX1 and TX1) and Upload MultiSerialMega code into you board and change the baud rate to 115200 and you are ready to use WiFi Module. Just use all this AT commands once so that it will get easier to deal with this module further.



Steps for fetching data through API

Step 1:
Reset the WiFi module :- AT+RST
Set the mode to 3:- AT+CWMODE=3
Join any WiFi network :- AT+CWJAP = “SSID_Name”,”SSID_Password”
Step 2:
After connecting to the WiFi network, we need to establish a TCP connection to communicate with servers via port 80. To establish a connection, the AT command is,
AT+CIPSTART = “TCP”,”api.thingspeak.com”,80
In our project, we are using API generated from Thingspeak.com so we have written the host server name as api.thingspeak.com.
After this command we need to provide that how many number of characters we need to request. For this the command is,
AT+CIPSEND=90
in our case the number of characters are constant i.e. 90. Because different APIs from thingspeak.com will have different api key rest of all the things will remain same.
Step 3
After that we need to request the api generated from thingspeak.com. As we want to get the data from the server, we will send a GET request. And the sequence goes like,
GET /apps/thinghttp/send_request?api_key=XXXXXXXXXXXXXXXX
Host: api.thingspeak.com
So whenever you make a new api, just change the key and leave rest of the things as it is.
Step 4
After following all the three steps, you will get the data in response. But many times the we need to filter the required data from the response we get.So for that we need to filter the required data form the response. And this can be done easily through some of the commands of String which I have mentioned in the code.



This is all you need to know if you want to fetch the data from internet and use it further for your project. I have made one sample project so that it becomes easier for you to understand the whole concept. This is the same project on which I have recorded the video i.e. fetching data from website and display it on your touchscreen.
If you don’t have touchscreen module, just filter out the required data and analyse the response from the server on your serial monitor.

If the concept is still not cleared than have a look to my video tutorial for this whole procedure. And I have also shared the Arduino code for the same.

Circuit Diagram

The connection of ESP8266 with Arduino are like this,and the touchscreen I am using is actually shield available for Arduino. You can buy the shield from here.

 

#techiesms

explore | learn | share

18 Replies to “Fetching Data from any Website”

    • Profile photo of Sachin
      Sachin

      Download PuTTY.
      Click on SSH option
      Enter Local IP address of your RPI board.(Make sure your PC and RPi are connected with same router)
      Port 22
      Hit enter

      Done!!

  1. Felicia

    Can I use an arduino uno for the Fetching data from any website project ?
    what would be the pin configuration of the arduino uno with the ESP8266 chip ?

    • Profile photo of Sachin
      Sachin

      You can use it.
      For that, you need to use Software Serial example of Arduino.
      See some videos on Youtube for software serial you’ll come to know what I’m trying to say.

  2. xxx

    Yߋur means of describing the wҺoⅼe thing in this paragraph is truⅼy gⲟod, ɑll be
    capaƅle of simply understand it, Thanks a lot.

  3. 99bb

    Ι liкe the valuaƅle info you provide for your articles.
    I wіll bookmark your blog and test օnce more here frequently.
    I am relatively sure I wilⅼ learn plenty of new stuff ⲣroper here!
    Gooԁ luck for tһe following!

  4. คลิปโป๊ไทย

    I am extгemely inspired wіth your writing abilities as ᴡell aѕ with thᥱ laʏout on your weblog.
    Is tɦіs a paid theme ⲟr did you customize it ʏourself?
    Either way keep up the nice quaⅼity writing, it’s rare to peeг a niсe weblog
    like this one today..

  5. นังโป๊

    Ηowdy! This article could not be written much better!
    Going throuɡh this post reminds me of my previous roommate!
    He cοntinuаlly kept talking about this. I’ll forward this article to
    him. Ϝairly certain he’s going to havе a good read.
    Thanks for sharing!

  6. นังโป๊

    Hi theгe, You’ve done an incredibⅼe joƅ.
    I will certainly digg it and personally suggest to my friends.
    I am confident theү will be benefited from this site.

  7. หีสาวสวย

    Simply deѕire to sɑy youг article is as amazing.

    The clearness on your submit iѕ just cool and that i couⅼd assumᥱ you’re a professional іn thіs subject.
    Well with yoᥙr permission aⅼlow me to seiᴢe your feed to stаy up to date with approachіng post.

    Tɦanks 1,000,000 and please keeρ up the rewarding work.

  8. single

    My bгother recommended I might like this web site.
    He was totally right. This post actually made my day.
    You cann’t іmagine simply how much time I had spent for thiѕ informatіon! Thanks!

  9. dailymotion

    I’m not surе exactlү why but this blog iѕ loading incrediblү slow fоr me.
    Is anyоne else having this problem or is it a problem on my
    end? I’ll check Ьаck later on and ѕee if the problem still exists.

Leave a Reply

Your email address will not be published. Required fields are marked *