Skip to main content

Using Twitter search API in Java

If you are reading this, I assume you are aware about Twitter APIs. If you are new to this API, I suggest you read the API documentation (which is quite good BTW!)

This simple program of mine uses Twitter search API to get "positive" tweets for a search string. It aims to retrieve all the search results (tweets) and hence has to make multiple calls to REST API, for which it uses max_id (explained later).

Let me give an overview of all the request parameters that we have specified in the URL (Twitter API URL):

q=%23SearchString: #SearchString 
count=100: Number of tweets you want the API to return in one call.
include_entities=false: Exclude details of entities in the JSON response. You can set it to true if you want.
max_id: Since we want more than 100 tweets, we are invoking the API multiple times and giving the id of last tweet returned in previous invocation as 'max_id' and asking Twitter to give 100 tweets prior to this tweet. 
e.g. First call returned tweets with TweetIDs 1000-900 (recent most first). In next call we set max_id = 900 and ask Twitter to send tweets with TweetIDs < 900.

We are doing this as long as response is null, i.e. no more tweets are available. I am using Application-only authorization for this example (since we don't have to search any particular user's timeline) and my motive is to find ALL the tweets.

Go ahead and give it a spin. Leave your questions in comments.
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
public class TwitterUtil {
    private static final String USER_AGENT = "Twitter Tutorial";
    private static int positiveCount;
    /**
     * @param args the command line arguments
     */
    public static void main(String[] args) {
        String key = "your twitter key";
        String secret = "twitter secret";
        String maxId = "";
        String concat = "";
        try {
            key = (URLEncoder.encode(key, "UTF-8"));
            secret = (URLEncoder.encode(secret, "UTF-8"));
            concat = key+":"+secret;
            byte[] encoded = Base64.encodeBase64(concat.getBytes());
//            System.out.println("Base64 Encoded String : " + new String(encoded));
            
            String bearerToken = "";
            
            try {
                bearerToken = generateBearerToken(new String(encoded));
                sendSearchRequest(bearerToken, maxId);
            } catch (Exception ex) {
                Logger.getLogger(TwitterUtil.class.getName()).log(Level.SEVERE, null, ex);
            }
            
            byte[] decoded = Base64.decodeBase64(encoded);      
            System.out.println("Base 64 Decoded  String : " + new String(decoded));
            
        } catch (UnsupportedEncodingException ex) {
            Logger.getLogger(TwitterUtil.class.getName()).log(Level.SEVERE, null, ex);
        }
        
        
            
    }
    
    private static String generateBearerToken(String encoded) throws Exception {
        String url = "https://api.twitter.com/oauth2/token";
        URL obj = new URL(url);
        HttpsURLConnection con = (HttpsURLConnection) obj.openConnection();

        //add reuqest header
        con.setRequestMethod("POST");
        con.setRequestProperty("User-Agent", USER_AGENT);
        con.setRequestProperty("Accept-Language", "en-US,en;q=0.5");
        con.setRequestProperty("Authorization", "Basic "+encoded);
        con.setRequestProperty("Content-Type", "application/x-www-form-urlencoded;charset=UTF-8");
               
        String urlParameters = "grant_type=client_credentials";

        // Send post request
        con.setDoOutput(true);
        DataOutputStream wr = new DataOutputStream(con.getOutputStream());
        wr.writeBytes(urlParameters);
        wr.flush();
        wr.close();

        int responseCode = con.getResponseCode();

        BufferedReader in = new BufferedReader(new InputStreamReader(con.getInputStream()));
        String inputLine;
        StringBuffer response = new StringBuffer();

        while ((inputLine = in.readLine()) != null) {
            response.append(inputLine);
        }
        in.close();

        //print result
//        String temp = new JSONObject(response.toString());
        JSONParser parser=new JSONParser();
        Object object = parser.parse(response.toString());
        JSONObject array = (JSONObject)object;
        String accessToken = (String)array.get("access_token");
        
        return (accessToken);
    }
    
    //This method sends request to Twitter API and is invoked recursively to get all tweets
    private static void sendSearchRequest(String token, String maxId) throws Exception {
        System.out.println("Entering sendSearchRequest");
        byte[] temp = Base64.encodeBase64(token.getBytes());
        String url = "https://api.twitter.com/1.1/search/tweets.json?q=%23MaryKom&count=100&include_entities=false";
        if(maxId != null)
            url += "&max_id="+maxId;
        URL obj = new URL(url);
        HttpsURLConnection con = (HttpsURLConnection) obj.openConnection();
        con.setRequestProperty("Authorization", "Bearer " + token);

        System.out.println("\nSending 'GET' request to URL : " + url);
        // optional default is GET
        con.setRequestMethod("GET");
        int responseCode = con.getResponseCode();
        
        System.out.println("Response Code : " + responseCode);

        BufferedReader in = new BufferedReader(
                new InputStreamReader(con.getInputStream()));
        String inputLine;
        StringBuffer response = new StringBuffer();

        while ((inputLine = in.readLine()) != null) {
            response.append(inputLine);
        }
        in.close();
        con.disconnect();
        

        if(response.toString().length() > 0){
            maxId = parseResponse(response.toString());
            System.out.println(positiveCount);
            Thread.sleep(20000);                       
            sendSearchRequest(token, maxId);
        }else{
            System.out.println("No more results");
            return;
        }
        //print result
        //System.out.println(response.toString());
        
        System.out.println("Exiting parseResponse");
    }

    //This method parses JSON response and returns ID of the last tweet.
    private static String parseResponse(String response) {
        System.out.println("Entering parseResponse");
        String lastId = "";
        JSONParser parser=new JSONParser();
        try{
            Object object = parser.parse(response);
            JSONObject array = (JSONObject)object;
            JSONArray jsonArray = (JSONArray)array.get("statuses");
            int size = jsonArray.size();
            positiveCount += size;
            System.out.println(size);
            
            JSONObject lastObject = (JSONObject)jsonArray.get(size-1);
             
//            System.out.println(lastObject);
            
            Long id = (Long)lastObject.get("id");
            System.out.println(id);
            lastId = id+"";
        }catch(Exception e){
            e.printStackTrace();
        }
        System.out.println("Exiting parseResponse");
        return lastId;
                
    }
    
}

Comments

Popular posts from this blog

File upload problem: UTF-8 encoding not honored when form has multipart/form-data

The problem that I was facing was something like this. I was using Apache Commons File Upload library to upload and download some file.

I had a form in which user can upload a file and another field 'name' in which she can give any name to the file being loaded.


When I submitted the form, the file was uploaded fine but the value in name field was garbled. I followed all the possible suggestions I found:

<%@page pageEncoding="UTF-8"%> set. <%@page contentType="text/html;charset=UTF-8"%gt; set after the first directive. <meta equiv="Content-Type" content="text/html;charset=UTF-8"> in the head. enctype="multipart/form-data" attribute in the form. accept-charset="UTF-8" attribute in the form.
in the Servlet:
before doing any operations on request object: request.setCharacterEncoding("UTF-8"); For accessing the value

FileItem item = (FileItem) iter.next();

if (item.isFormField()) {

//For regular…

java.lang.IllegalArgumentException: Malformed \uxxxx encoding

I was getting this exception during build while running ant. Googling didn't help much and I was flummoxed because the same code was running fine till now.

My code reads a text file and does some operations on the basis of values read. It was only when I saw the text files I understood the error. I had copied the text in wordpad and saved it as .txt file. Wordpad had put lot of formatting information before and after the content. Also there was "\par" after every line, which was giving this error.

So moral of the story: if you get this exception check your properties file (or any other file that your code might be reading.)

Easiest way to print Timestamp in Java

Rather than using Calendar.getTime() we can use java.sql.Timestamp class to get the time stamp which gives date and time till millisecond precision.

System.out.println(new Timestamp(System.currentTimeMillis()));

Above will give you current timestamp in this format: 2010-07-27 16:37:45.39