Skip to main content

Using Twitter search API in Java

If you are reading this, I assume you are aware about Twitter APIs. If you are new to this API, I suggest you read the API documentation (which is quite good BTW!)

This simple program of mine uses Twitter search API to get "positive" tweets for a search string. It aims to retrieve all the search results (tweets) and hence has to make multiple calls to REST API, for which it uses max_id (explained later).

Let me give an overview of all the request parameters that we have specified in the URL (Twitter API URL):

q=%23SearchString: #SearchString 
count=100: Number of tweets you want the API to return in one call.
include_entities=false: Exclude details of entities in the JSON response. You can set it to true if you want.
max_id: Since we want more than 100 tweets, we are invoking the API multiple times and giving the id of last tweet returned in previous invocation as 'max_id' and asking Twitter to give 100 tweets prior to this tweet. 

e.g. First call returned tweets with TweetIDs 1000-900 (recent most first). In next call we set max_id = 900 and ask Twitter to send tweets with TweetIDs < 900.



We are doing this as long as response is null, i.e. no more tweets are available. I am using Application-only authorization for this example (since we don't have to search any particular user's timeline) and my motive is to find ALL the tweets.



Go ahead and give it a spin. Leave your questions in comments.






  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156


public class TwitterUtil {
    private static final String USER_AGENT = "Twitter Tutorial";
    private static int positiveCount;
    /**
     * @param args the command line arguments
     */
    public static void main(String[] args) {
        String key = "your twitter key";
        String secret = "twitter secret";
        String maxId = "";
        String concat = "";
        try {
            key = (URLEncoder.encode(key, "UTF-8"));
            secret = (URLEncoder.encode(secret, "UTF-8"));
            concat = key+":"+secret;
            byte[] encoded = Base64.encodeBase64(concat.getBytes());
//            System.out.println("Base64 Encoded String : " + new String(encoded));
            
            String bearerToken = "";
            
            try {
                bearerToken = generateBearerToken(new String(encoded));
                sendSearchRequest(bearerToken, maxId);
            } catch (Exception ex) {
                Logger.getLogger(TwitterUtil.class.getName()).log(Level.SEVERE, null, ex);
            }
            
            byte[] decoded = Base64.decodeBase64(encoded);      
            System.out.println("Base 64 Decoded  String : " + new String(decoded));
            
        } catch (UnsupportedEncodingException ex) {
            Logger.getLogger(TwitterUtil.class.getName()).log(Level.SEVERE, null, ex);
        }
        
        
            
    }
    
    private static String generateBearerToken(String encoded) throws Exception {
        String url = "https://api.twitter.com/oauth2/token";
        URL obj = new URL(url);
        HttpsURLConnection con = (HttpsURLConnection) obj.openConnection();

        //add reuqest header
        con.setRequestMethod("POST");
        con.setRequestProperty("User-Agent", USER_AGENT);
        con.setRequestProperty("Accept-Language", "en-US,en;q=0.5");
        con.setRequestProperty("Authorization", "Basic "+encoded);
        con.setRequestProperty("Content-Type", "application/x-www-form-urlencoded;charset=UTF-8");
               
        String urlParameters = "grant_type=client_credentials";

        // Send post request
        con.setDoOutput(true);
        DataOutputStream wr = new DataOutputStream(con.getOutputStream());
        wr.writeBytes(urlParameters);
        wr.flush();
        wr.close();

        int responseCode = con.getResponseCode();

        BufferedReader in = new BufferedReader(new InputStreamReader(con.getInputStream()));
        String inputLine;
        StringBuffer response = new StringBuffer();

        while ((inputLine = in.readLine()) != null) {
            response.append(inputLine);
        }
        in.close();

        //print result
//        String temp = new JSONObject(response.toString());
        JSONParser parser=new JSONParser();
        Object object = parser.parse(response.toString());
        JSONObject array = (JSONObject)object;
        String accessToken = (String)array.get("access_token");
        
        return (accessToken);
    }
    


    //This method sends request to Twitter API and is invoked recursively to get all tweets
    private static void sendSearchRequest(String token, String maxId) throws Exception {
        System.out.println("Entering sendSearchRequest");
        byte[] temp = Base64.encodeBase64(token.getBytes());
        String url = "https://api.twitter.com/1.1/search/tweets.json?q=%23MaryKom&count=100&include_entities=false";
        if(maxId != null)
            url += "&max_id="+maxId;
        URL obj = new URL(url);
        HttpsURLConnection con = (HttpsURLConnection) obj.openConnection();
        con.setRequestProperty("Authorization", "Bearer " + token);

        System.out.println("\nSending 'GET' request to URL : " + url);
        // optional default is GET
        con.setRequestMethod("GET");
        int responseCode = con.getResponseCode();
        
        System.out.println("Response Code : " + responseCode);

        BufferedReader in = new BufferedReader(
                new InputStreamReader(con.getInputStream()));
        String inputLine;
        StringBuffer response = new StringBuffer();

        while ((inputLine = in.readLine()) != null) {
            response.append(inputLine);
        }
        in.close();
        con.disconnect();
        

        if(response.toString().length() > 0){
            maxId = parseResponse(response.toString());
            System.out.println(positiveCount);
            Thread.sleep(20000);                       
            sendSearchRequest(token, maxId);
        }else{
            System.out.println("No more results");
            return;
        }
        //print result
        //System.out.println(response.toString());
        
        System.out.println("Exiting parseResponse");
    }




    //This method parses JSON response and returns ID of the last tweet.
    private static String parseResponse(String response) {
        System.out.println("Entering parseResponse");
        String lastId = "";
        JSONParser parser=new JSONParser();
        try{
            Object object = parser.parse(response);
            JSONObject array = (JSONObject)object;
            JSONArray jsonArray = (JSONArray)array.get("statuses");
            int size = jsonArray.size();
            positiveCount += size;
            System.out.println(size);
            
            JSONObject lastObject = (JSONObject)jsonArray.get(size-1);
             
//            System.out.println(lastObject);
            
            Long id = (Long)lastObject.get("id");
            System.out.println(id);
            lastId = id+"";
        }catch(Exception e){
            e.printStackTrace();
        }
        System.out.println("Exiting parseResponse");
        return lastId;
                
    }
    
}






















Labels:



































Comments








Post a Comment
























Popular posts from this blog



















How to upload to Google Cloud Storage buckets using CURL 




























 Signed URLs are pretty nifty feature given by Google Cloud Platform to let anyone access your cloud storage (bucket or any file in the bucket) without need to sign in.     Official documentation  gives step by step details as to how to read/write to the bucket using gsutil or through a program. This article will tell you how to upload a file to the bucket using curl so that any client which doesn't have cloud SDK installed can do this using a simple script.      This command creates a signed PUT URL for your bucket.      gsutil signurl -c 'text/plain' -m PUT serviceAccount.json gs://test_bucket_location      Here is my URL:   https://storage.googleapis.com/test_sl?GoogleAccessId=my-project-id@appspot.gserviceaccount.com&Expires=1490266627&Signature=UfKBNHWtjLKSBEcUQUKDeQtSQV6YCleE9hGG%2BCxVEjDOmkDxwkC%2BPtEg63pjDBHyKhVOnhspP1%2FAVSr%2B%2Fty8Ps7MSQ0lM2YHkbPeqjTiUcAfsbdcuXUMbe3p8FysRUFMe2dSikehBJWtbYtjb%2BNCw3L09c7fLFyAoJafIcnoIz7iJGP%2Br6gAUkSnZXgbVjr6wjN%2FIaudXIqA




















Post a Comment









Read more






























Running Apache Beam pipeline using Spark Runner on a local standalone Spark Cluster




























 The best thing about Apache Beam ( B atch + Str eam ) is that multiple runners can be plugged in and same pipeline can be run using Spark, Flink or Google Cloud Dataflow.   If you are a beginner like me and want to run a simple pipeline using Spark Runner then whole setup may be tad daunting.   Start with Beam's WordCount examples  which help you quickstart with running pipelines using different types of runners. There are code snippets for running the same pipeline using different types of runners but here the code is running on your local system using Spark libraries which is good for testing and debugging pipeline. If you want to run the pipeline on a Spark cluster you need to do a little more work!   Let's start by setting up a simple standalone single-node cluster on our local machine. Extending the cluster is as easy as running a command on another machine, which you want to add to cluster.   Start with the obvious: install spark on your machine! (Remember to have Java a




















Post a Comment









Read more




























java.lang.IllegalArgumentException: Malformed \uxxxx encoding




























I was getting this exception during build while running ant. Googling didn't help much and I was flummoxed because the same code was running fine till now. My code reads a text file and does some operations on the basis of values read. It was only when I saw the text files I understood the error. I had copied the text in wordpad and saved it as .txt file. Wordpad had put lot of formatting information before and after the content. Also there was "\par" after every line, which was giving this error. So moral of the story: if you get this exception check your properties file (or any other file that your code might be reading.)




















7 comments









Read more