Get Your Duolingo Stats

Doulingo is one of the most popular applications for learning a new language. Duolingo has a good selection of languages and it’s free.

If you wanted to create your apps to look at your progress or do some comparisons with other Doulingo users it’s a little challenging because there isn’t a documented API (Application Programming Interface).

In this blog I wanted to look at two programming choices that are available. The first is a Web page (REST API) query and the second is a Python library.

It is important to note that last year the Doulingo interface changed so a lot of the documentation on the Internet is not 100% valid.

The Web Page (REST API) Query

If you enter : https://www.duolingo.com/users/a_user_name

You will see a very large amount of JSON (JavaScript Object Notation) text. This undocumented text is quite ugly.

web_restapi

It is possible to use some Python code to call the Web page, and try to make the JSON a little more readable. The code below uses the requests library to get the Web data, and the json library to format the web output (JSON) data.

import requests
import json

r = requests.get('https://www.duolingo.com/users/some_users')

jtext = json.loads(r.text)

with open('user_data.txt', 'w', encoding='utf8') as outfile:
     json.dump(jtext, outfile, sort_keys = False, indent = 4,
               ensure_ascii = False)

This example created an outfile that is much easier (but far from simple) to analyze. Below is a section of the output file where we can see the languages section broken out.

    "tts_base_url_http": "http://d7mj4aqfscim2.cloudfront.net/",
    "id": 141898265,
    "dict_base_url": "http://d2.duolingo.com/",
    "cohort": null,
    "daily_goal": 10,
    "delete-permissions": false,
    "ads_enabled": true,
    "languages": [         {
            "streak": 4,
            "language_string": "Greek",
            "points": 0,
            "learning": false,
            "language": "el",
            "level": 1,
            "current_learning": false,
            "sentences_translated": 0,
            "to_next_level": 60
        },
        {
            "streak": 4,
            "language_string": "Esperanto",
            "points": 0,
            "learning": false,
            "language": "eo",
            "level": 1,
            "current_learning": false,
            "sentences_translated": 0,
            "to_next_level": 60
        },

Show the Languages and Work History

By looking at the outfile the languages studied and the work history can be determined:

import requests
import json
from datetime import datetime

r = requests.get('https://www.duolingo.com/users/pete_xxxxx')

j = json.loads(r.text)

print ("Languages Studied\n")
for thelang in j["languages"]:
    if thelang["points"] > 0 :
        print (thelang["language_string"], ",Level = ", thelang["level"], ",Points=", thelang["points"])

print ("\n\nWork History\n")

for ical in j["calendar"]:
    ddate =  datetime.fromtimestamp(ical["datetime"] / 1e3)
    sdate = ddate.strftime("%m/%d/%Y")
    print(sdate, ", Improvement = ", ical['improvement'])

For my example this gave the following output:


Languages Studied

Indonesian ,Level = 6 ,Points= 570
Spanish ,Level = 13 ,Points= 5102
Chinese ,Level = 2 ,Points= 70
French ,Level = 9 ,Points= 1710
German ,Level = 6 ,Points= 616
Japanese ,Level = 10 ,Points= 2297

Work History

02/22/2019 , Improvement = 10
02/22/2019 , Improvement = 10
02/23/2019 , Improvement = 10
02/23/2019 , Improvement = 10
02/24/2019 , Improvement = 10

Duolingo Python Library

There is an unofficial Duolingo Python Library, the author did an excellent job of putting together some useful calls. Unfortunately since the Duolingo interface changed last year not all the functions are now available or working. However the key ones are still usable.

To install the library go to a terminal window or command prompt and enter:

pip install duolingo-api

A Python example would be:

import duolingo
from datetime import datetime

lingo  = duolingo.Duolingo('pete_xxxxx')

# get_user_info : works but outputs too much info
#print (lingo.get_user_info())

print ("Languages I am studying:")
print (lingo.get_languages(abbreviations=False))

print("\n\nLanguage Progress\n")
for ilang in lingo.get_languages(abbreviations=False):
       jlang = lingo.get_language_details(ilang)
       print( jlang["language_string"]," ,Points: ",jlang["points"], "Level = ", jlang["level"])

print("\n\nWork History\n")
for ical in lingo.get_calendar():
    ddate =  datetime.fromtimestamp(ical["datetime"] / 1e3)
    sdate = ddate.strftime("%m/%d/%Y")
    print(sdate, ", Improvement = ", ical['improvement'])

For my example the output is:


Languages I am studying:
['Indonesian', 'Spanish', 'Chinese', 'French', 'German', 'Japanese']

Language Progress

Indonesian ,Points: 570 Level = 6
Spanish ,Points: 5102 Level = 13
Chinese ,Points: 70 Level = 2
French ,Points: 1710 Level = 9
German ,Points: 616 Level = 6
Japanese ,Points: 2297 Level = 10

Work History

02/22/2019 , Improvement = 10
02/22/2019 , Improvement = 10
02/23/2019 , Improvement = 10
02/23/2019 , Improvement = 10
02/24/2019 , Improvement = 10

Summary

Now that we have a basic interface to Duolingo it’s possible to start doing some trends and charts. Below a is some simple chart code for language levels.

import duolingo
import numpy as np
import matplotlib.pyplot as plt

lingo  = duolingo.Duolingo('pete_the_dude')

thelang = lingo.get_languages(abbreviations=False)

thelevel = []
for ilang in lingo.get_languages(abbreviations=False):
       jlang = lingo.get_language_details(ilang)
       thelevel.append(jlang["level"])


plt.bar(thelang,thelevel, color='r')

plt.legend()
plt.xlabel('Language')
plt.ylabel('Level')

plt.title('My Duolingo Progress')

plt.show()

Duo_progress

Javascript talking to littleBits Cloud API

LittleBits is a set of components that allow kids to build their own electrical circuits. The Cloudbit will send and receive values from the Internet.

For this project we wanted to build some Web pages with Javascript that could interact with the littleBits Cloud API.

littleBits Hardware Setup

cloudbit_setup

The Cloudbit enables an input to be sent to the Internet, and it will receive a value (output) from the Internet. For a test setup we used a dimmer bit to adjust the input value with a number bit to show the value. On the output side we used a bargraph bit. Other combinations of bits are possible. The key is to be able to test and see your input and output values.

Cloudbits Setup

For the Cloudbit setup you will need to create a sign-in on the littleBits web site. For details on how to setup your Cloudbit use the following link.

The Cloud API is a RESTful interface, that uses http requests with some header definitions. Before you get started you’ll need your specific littleBits device ID and authorization token. In your specific Cloudbit definition go to the Settings icon, and get your Device ID and AccessToken.

littlebit_id

Cloudbit Rest API – First Example Testing with curl

The Cloudbit API reference can be found at : http://developers.littlebitscloud.cc/#devices. 

The Cloudbit API is based on the REST API standard, and this means that an HTTP request is made with some extra parameters that are passed in the header. The most important header item is the littleBits authorization token.

Curl is command line software tool that allows you to issue an HTTP request with header and method parameters and it returns the results in a text format. Curl is available for Windows, OSX and Linux. As an example to query the CloudBit API to get all my devices the following command is issued (note you’ll need to enter your own authorization code) :

curl "https://api-http.littlebitscloud.cc/v2/devices" \
-H "Authorization: Bearer 4f3830b44e1d4b2789b50b0xxxxxx"

[
{"label":"twinbit",
"id":"00e04c0379bb",
"subscriptions":[],
"subscribers":[],
"user_id":118217,
"is_connected":true,
"input_interval_ms":200}
]

Javascript Device Monitor

The curl example can be done in Javascript using the Javascript built in XMLHttpRequest object.

A monitor page can be useful if you have a number of Cloudbits and you’re interested is checking their status. Below is a sample web page and the associated HTML/javascript code. It’s important to note that text that is returned needs to have the first few characters removed in order to have a clean JSON string.

js_status

<!DOCTYPE html>
<html>
<body>
<h1 id='title'>Cloudbit Status Monitor</h1>
<h2>
Label : <font color='red' id='LB_label'></font><br>
Is Connected : <font color='red' id='is_connected'></font><br>
</h2>
 
<script>
var xhttp = new XMLHttpRequest();
// change the authtoken to match your settings
var authtoken = "4f3830b44e1dxxxxxxxxxxxxxxxx";
var theURL =  "https://api-http.littlebitscloud.cc/devices/";
xhttp.open("GET", theURL, true);
xhttp.setRequestHeader("Accept","application/vnd.littlebits.v2+json");
xhttp.setRequestHeader("Authorization", "Bearer " + authtoken);
xhttp.onreadystatechange = function() {
  if (xhttp.readyState == 4 ) { // when the response is complete get the data
    // remove leading "[" and trailing ']'
    var theresponse = xhttp.responseText.substring(1, xhttp.responseText.length -1);
    var lb_data = JSON.parse( theresponse );  
    
    document.getElementById('LB_label').innerText =  lb_data.label;
    document.getElementById('is_connected').innerHTML = lb_data.is_connected;
  }
}
xhttp.send();
</script>

</body>


</html>

Read Value Example

Reading a CloudBit value is a little tricky because the data is returned as a stream that updates every second. I saw a lot of people asking for a single point read, so hopefully this will be available soon.

The read value HTTP request is a little more complex than the simple monitor device example because you need to include the device that you are querying and a head parameter of :  Accept: application/json.

A curl example would be:

curl -XGET "https://api-http.littlebitscloud.cc/v2/devices/00e04c0379bb/input" -H "Authorization: Bearer 4f3830b44e1d4b2789b50b09cb493f06750b968cff5d45331c75006025fa0dc9" -H Accept: application/vnd.littlebits.v2+json"

data:{"type":"input","timestamp":1543536039623,"percent":6,"absolute":66,"name":"amplitude","payload":{"percent":6,"absolute":66},"from":{"device":{"id":"00e04c0379bb","mac":"00e04c0379bb"}}}

data:{"type":"input","timestamp":1543536039650,"percent":6,"absolute":65,"name":"amplitude","payload":{"percent":6,"absolute":65},"from":{"device":{"id":"00e04c0379bb","mac":"00e04c0379bb"}}}

...

For an HTML/Javascript single point read the key to check the xhttp.readyState == 3 , this will catch the first streamed response.  Below is a single point example with code.

js_in

<!DOCTYPE html>
<html>
<head>
<title>littleBits Get Input</title>
<script>
var theinput = 0;

function get_input() {
// update with your deviceid and authtoken
  var deviceid = "00e04c0379bb";
  var authtoken = "4f3830b44e1d4b27xxxx";
  var theurl = "https://api-http.littlebitscloud.cc/devices/";
  var xhttp = new XMLHttpRequest();
  xhttp.onreadystatechange = function() {

    if (xhttp.readyState == 3 ) {
	  	var datapackage = xhttp.responseText.split("\n\ndata:");
		var lb_data = JSON.parse( datapackage[1] );		 
		document.getElementById("thevalue").innerText =  lb_data.percent;
		xhttp.open("GET","",true);
		xhttp.send();
    }
  }
  xhttp.open("GET", theurl + deviceid + "/input", true);
  xhttp.setRequestHeader("Accept","application/json");
  xhttp.setRequestHeader("Authorization", "Bearer " + authtoken);
  xhttp.send();
}
</script>
</head>
<body>

<h1 id='title'>littleBit Get Input</h1>
The value : <font id="thevalue"> XXXX </font>
<button type="button" onclick="get_input()">Request data</button>
<br>

</body>
</html>

Output Example

A CloudBit output can be either sustained or it can be timed out. The curl command is:

curl "https://api-http.littlebitscloud.cc/v2/devices/yourdevice_id/output" \
  -X POST \
  -H "Authorization: your_auth_code" \
  -H "Content-type: application/json" \
  -d '{ "percent": 100, "duration_ms": 3000 }'

For an HTML/Javascipt web the http response is a POST, and the value’s percent and duration are put into a JSON string:

var params = JSON.stringify({duration_ms: duration,percent: thevalue});

An example page and code is shown below:

js_out

<!DOCTYPE html>
<html>
<body>
<h2>CloudBit Output Test</h2>
<button type="button" onclick="sendoutput()">Send Output</button>
<br>
<pre>
Output Time (ms): <input type="text" id="duration" value="-1"/> (constant = -1)</br>
Output Value	: <input type="text" id="thevalue" value="80"/> (percent 0-100)</br> 
</pre>
<p id="demo"></p>


function sendoutput() {
	var xhttp = new XMLHttpRequest();
        // change to your device id
	xhttp.open("POST", "https://api-http.littlebitscloud.cc/devices/00e04c0379bb/output?", true);
	xhttp.setRequestHeader("Accept","application/vnd.littlebits.v2+json");
	// change to your Auth Token
	xhttp.setRequestHeader("Authorization", "Bearer 4f3830b44e1d4b27xxxx");
	xhttp.setRequestHeader("Content-Type","application/json");

	var duration = document.getElementById("duration").value;
	var thevalue = document.getElementById("thevalue").value;

	var params = JSON.stringify({duration_ms: duration,percent: thevalue});

	xhttp.onreadystatechange = function() {
		document.getElementById("demo").innerHTML = "Result=" +xhttp.responseText;
	}

	xhttp.send(params);
}


</body>
</html>

Gauges

Once you’ve got the basics down it’s possible to start making some more advanced applications. There are lot of Javascript charting libraries. For this example I used the Google  Charts library.

js_gauge

<html>
  <head>
   https://www.gstatic.com/charts/loader.js
   
	  var littleBitinput = 0;
      google.charts.load('current', {'packages':['gauge']});
      google.charts.setOnLoadCallback(drawChart);
      function drawChart() {

        var data = google.visualization.arrayToDataTable([
          ['Label', 'Value'],
          ['littleBits', 100],
        ]);

        var options = { 
          width: 600, height: 600,
          redFrom: 90, redTo: 100,
          yellowFrom:75, yellowTo: 90,
		  greenFrom: 60, greenTo: 75,
          minorTicks: 5
        };

        var chart = new google.visualization.Gauge(document.getElementById('chart_div'));

        chart.draw(data, options);

        setInterval(function() {
		  get_input();
          data.setValue(0, 1, littleBitinput);
          chart.draw(data, options);
        }, 500);
 
      }
	  function get_input() {
	  // change to your deviceid and authtoken
		  var deviceid = "00e04c0379bb";
		  var authtoken = "4f3830b44e1d4b27xxx";
		  var theurl = "https://api-http.littlebitscloud.cc/devices/";		  
		  var xhttp = new XMLHttpRequest();
		  
		  xhttp.onreadystatechange = function() {
			if (xhttp.readyState == 3 ) {
				var datapackage = xhttp.responseText.split("\n\ndata:");
				var lb_data = JSON.parse( datapackage[1] );		 
				littleBitinput =  lb_data.percent;
				xhttp.open("GET","",true);
				xhttp.send();
			}
		  }
		  xhttp.open("GET", theurl + deviceid + "/input", true);
		  xhttp.setRequestHeader("Accept","application/vnd.littlebits.v2+json");
		  xhttp.setRequestHeader("Authorization", "Bearer " + authtoken);
		  xhttp.send();

}
    
  </head>
  <body>

 

Final Comment

There are some great projects that you could do with littleBits and the CloudAPI.

One still project I did was to use smoothie charts http://smoothiecharts.org/ for the real time charts.

lb_chart2