Liberating Your Data from the Basis B1 Band

Basis B1After months of being on the waiting list to get my hands on the sweet-looking Basis B1 band, I ran out of patience and picked one up off of EBay (at a premium of $100 more than it’s $199 retail price).
For those of you who aren’t familiar with the Basis B1 band, while it looks like a digital watch at first glance, it is built around five sensors that work together to track various aspects of your health:

  • Optical blood flow sensor – This sensor detects heart rate, through pulse or blood flow, and constantly monitors heart rate. This is the most important sensor and provides the device with key metrics so that it can understand how the body is reacting to lifestyle
    events.
  • 3D accelerometer – Highly sensitive, this sensor detects even the smallest movement, regardless of whether users are alert and active or sleeping soundly.
  • Body temperature – By measuring the user’s skin surface temperature, this sensor provides a more accurate reading of exertion in an activity state.
  • Ambient temperature reading – This sensor detects the outside temperature and, in relation to body temperature, can boost the accuracy of caloric burn calculations, as well as providing insights into other environmental effects on the body.
  • Galvanic skin response – By measuring the intensity of the skin’s moisture output (or sweat) during any activity, from warm-up to recovery, this sensor provides information particular to an individual’s level of exertion when engaged in activity.

Basis dashboardI’ve been using it non-stop for the past few weeks and in a nutshell, while it’s still very much “beta”, it’s a great device with a very promising future, and I’m looking forward to firmware enhancements to improve it’s display functions and enabling bluetooth support (which it has but currently is disabled) to allow for wireless syncing and pairing with other apps (i.e., HRV, etc.). While you are able to sync the data off of your B1 band to Basis’ website (via a USB cable – again, bluetooth support coming in the future) and they generate some neat graphs, like any dedicated self-tracker, I was disappointed that Basis doesn’t provide a way for me to download my raw data. According to Basis literature:

While much of this data and the individual metrics collected by sensors are not visible to the user, they are aggregated and summarized by Basis B1 to build to provide a complete picture of the person’s health.

Boo. But not one to be discouraged, after some digging around I was able to figure out how to get my data, and wrote a data export script that I’ve made publicly available on GitHub.

Basis currently returns the following data points. They will represent an average (for heart rate) or sum (steps) over the previous 1-minute period:

  • Time – time reading was taken
  • Heart Rate – beats per minute
  • Steps – number of steps taken
  • Calories – number of calories burned
  • GSR – Galvanic skin response (i.e., sweat/skin conductivity). Learn more about GSR here – http://en.wikipedia.org/wiki/Skin_conductance
  • Skin Temperature – skin temperature (degrees F)
  • Air Temperature – air temperatute (degrees F)

There are some other aggregate metrics included in the response such as min/max/average/standard deviation metrics for each set of data.

Basis summary and body statesSummary and Body State Data Export

Basis sensor data exportBasis B1 Sensor Data Export

For a more in-depth overview of the script (and to download it for your own use, please visit my GitHub page!

Learn  more about Basis.

Read the Basis Technology Overvew.

 

  • http://biohacked.net Damien

    Thanks for this – I’ll be getting mine in a few days, looking forward to getting it set up for HRV.

  • http://profile.typepad.com/6p017d3d7cbf3f970c Bob T

    Hi Damien! Unfortunately, I don’t think the Basis B1 is saving heartrate data at a high enough sampling rate to do HRV. They seem to be storing 1 reading per minute. Check out this post on the Quantified Self forums from their former CTO/co-founder:
    https://forum.quantifiedself.com/thread-emwave2-heart-rate-variability-monitor?pid=613#pid613
    However, once bluetooth is enabled, would be great if it could simply transmit data to an HRV app running on my iPhone (or Android device).

  • http://biohacked.net Damien

    Hey Bob,
    Yeah, it’s unfortunate about HRV, I was looking forward to getting rid of the inconvenience of carrying the Emwave2 attached to my ear and getting all day readings.
    I see the mybasis only tracks signals as an average per minute, I hope that isn’t a hardware limitation and can be upgraded with firmware later. Would be invaluable. I’ll be looking for that bluetooth workaround too.
    I’ve installed your script and got my device today – looking forward to see the results!

  • http://profile.typepad.com/edrabbit1 Edrabbit

    Thanks so much for the tool!!
    If you want to archive a large number of data you can use curl to do it easily:
    For example to get all of May cached in /data:
    curl http://www.path.to/basisdataexport.php?date=2013-05-01-31

  • http://profile.typepad.com/6p017d3d7cbf3f970c Bob T

    Great tip! I’ll add this to the README file (and give you credit for it :)

  • Nathanael

    I really want to get my B1 data, but I’m a newbie to running scripts, etc. I’m not sure I have an executable location or where my webserver document root is. Can anyone give me a little more direction or point me to a site that would walk me through this? I would be grateful.

  • http://profile.typepad.com/6p017d3d7cbf3f970c Bob T

    Hi Nathaneal,
    Yeah, sorry but it does require a little bit of coding knowhow. If you are on a Mac you can just pop the script in your webserver root folder, configure as outlined above using any text editor, then access via a web browser.
    You can learn more about setting things up here:
    http://osxdaily.com/2012/09/02/start-apache-web-server-mac-os-x/

  • http://profile.typepad.com/6p017d3d7cbf3f970c Bob T

    Damien, I recently came across another company/device called Empatica that looks very promising and could provide real-time HRV. Unfortunately, doesn’t look like they are in-market yet:
    https://www.empatica.com/

  • Nathanael

    OK. So I figured out how to run the script and even use CURL. Now how to convert the json files to excel files?

  • Nathanael

    Thanks. I talked to a program friend and he hooked me up with winamp and then I installed curl. Now on to getting the data out of json into excel so that I can actually do something with it. Any help there?
    Also I’d be interested in hearing what sort of analysis folks have done once they get this great boatload of data. I have some thoughts about looking for times of day and days of the week that I’m most consistent and most in need of improvement.

  • JonnyBones

    Nice script, thanks for posting! Just trying to get Apache going my Linux box so I can use it. Personally, I’m planning to convert JSON to python data and then do data fusion with a whole bunch of other data sources I have – bicycle power meter data, Zigbee sensor data, GPS position. Thanks for unlocking a really cool piece of hardware.

  • http://profile.typepad.com/6p017d3d7cbf3f970c Bob T

    Hi Nathaneal,
    Yeah sorry I’ve been planning on adding an option to allow for csv export vs. json. In the meantime, there are a number of utilities that will do the conversion, such as http://www.danmandle.com/blog/json-to-csv-conversion-utility/

  • http://profile.typepad.com/6p017d3d7cbf3f970c Bob T

    Very cool! I’m working on some similar data integrations as well. It might be easier for you to just port my entire script to python so you can do the data retrieval/parsing/storage all at once.

  • Oddjob Irony

    What you’re doing will basically work, but there are a few issues:
    1) The number you are using in your API calls is not a user ID, it’s an OAuth token. These expire. They’re also a lot easier to find in cookies/local storage. Ideally, your script would authenticate to get its own.
    2) The data you’re getting back is not from the sensors–it’s been heavily processed. If you want raw sensor output, you’d have to wireshark it out during the upload or use a USB bus analyzer or something.
    3) That’s at least 20 times as much code as you actually need to do this.

  • Oddjob Irony

    Oops, I take that back. That string **IS** the user ID, now.
    If they’re sending your script that data without a valid OAuth token, then I’d say you’ve just discovered a massive security hole :)

  • http://profile.typepad.com/6p017d3d7cbf3f970c Bob T

    Thanks for the feedback! However, to address your points:
    1) I believe you are incorrect about OAuth. Since I’m hitting their API via a public endpoint (i.e., that’s how their charts make requests to grab user data), there is no OAuth token – that *is* a user id (which seems to follow MogoDB conventions).
    2) I totally agree about ideally getting raw sensor data, but that would require way more effort than I have time for, and the data is transmitted over SSL. I tried out Wireshark before writing this script, but gave up as I couldn’t get it to work.
    3) Totally agree – the script is a quick hack that works, but I haven’t spent much time optimizing it. Feel free to check out the code via it’s GitHub repo and polish it up! :)

  • http://profile.typepad.com/6p011570a5c85e970c Beau Gunderson

    Hey Bob, check out my progress in reverse engineering the raw data format here:
    https://github.com/beaugunderson/re-basis

  • http://profile.typepad.com/6p017d3d7cbf3f970c Bob T

    Beau, this looks very cool! How are you capturing the raw data? Could you add more info to your Readme file explaining what your script is doing? Thanks!

  • http://profile.typepad.com/6p011570a5c85e970c Beau Gunderson

    Captured by USBPcap and Wireshark on my Windows machine:
    http://desowin.org/usbpcap/
    Then reformatted using reformat.py (which just removes the interleaved USB packet headers) and then that’s fed into my analysis/reverse engineering attempts in inspect.py.
    Will add that to the README–hoping to give it some more publicity to find other people interested in attempting to figure out the data format!

  • http://profile.typepad.com/6p017d3d7cbf3f970c Bob T

    Well done! I’d be curious to know what level of granularity the USB packets contain – using my workaround, I can only get 1-minute intervals but ideally would like 1-second (or less) readings.

  • JonnyBones

    Actually, you’re right, it is easier to make a port. Sadly, I’ve got zero PHP experience, but I figured out what your script is doing overall. Just working on my python version now. Never would have figured out how to do this without your good documentation. Thanks again.

  • http://profile.typepad.com/6p017d3d7cbf3f970c Bob T

    Let me know if/when you’ve got your python script working and I’ll post a link to it. I agree that it might actually be easier to do in python if you are just running the script command-line as I’ve got some browser-side output that I’ve been using more for debugging than anything else.

  • http://personal.customerservant.com Amanda

    Hi Bob,
    I just got my Basis a few days ago, and am playing around with your script. I’m using it to look at my data since there are accessibility with screen reader issues on the main site. Thanks for providing it. I think the only thing I need to figure out is why the time and date functions aren’t working properly. But, huge thanks again.

  • http://personal.customerservant.com Amanda

    Oh, and enjoying the rest of your site as well, especially all the experiments you’re running. Also followed you on Twitter.

  • Nadir

    Hey guys
    this looks really cool. I am also interested to get the raw data (mainly the accelerometeer, but of course having the other would be awesome) to infer the kind of activity people are doing. There is some work out there that has been investigating activity recognition using 3D accelerometer, and I would like to experiment with the data recorded by the basis B1.
    Do you know if there has been any progress in getting this data?
    Thanks a lot
    Nadir

  • http://profile.typepad.com/6p017d3d7cbf3f970c Bob T

    Hi Amanda! Glad to hear you are finding the script useful. Not sure what you mean about the time/date functions not working properly?

  • http://profile.typepad.com/6p017d3d7cbf3f970c Bob T

    Hi Nadir! I haven’t yet been able to figure out how to get the raw accelerometer data off of my Basis band. I have a feeling that data is never sent to Basis’ servers (it’s used/stored on the device to calculate ‘steps’), so there would be no API endpoint. Unfortunately, looks like your only option would be to directly access the data contained on the band itself. :(

  • Jay

    Hi Bob! Accessing the data directly is what interests me the most. I’ll pay extra to keep the data in my control, without the need to upload it. Bob/Beau, any thoughts on the likelihood of being able to intercept the data and keep it locally, and only locally?

  • https://github.com/jonnybones/pybasis JonnyBones

    I did get my python script running, which turned out to be super easy. I ended up going off onto tangents with my other data and never did publish what I did. Shame on me..
    Anyway, I’ve put the code on GitHub at https://github.com/jonnybones/pybasis in case anyone is interested.
    Also, I upgraded my watch to Firmware 2.6 which tries to track running / walking / biking. Now I want to get that data too. :-)
    In the network viewer of my browser, I see a request for a JSON object with a URL like https://app.mybasis.com/api/v2/users/me/days/2013-12-29/activities?expand=activities&type=run,walk,bike . It does contain info about when biking happens, etc. So far so good..
    However, I get ‘401: Unauthorized’ if I try open that URL with Python. Not sure how the authentication works, but I suspect that the process is not as simple as before. Any ideas? Thanks!

  • http://profile.typepad.com/volchenboum Sam Volchenboum

    Any progress in getting at the raw data? I’ve been playing with the scripts, and it is easy to get the data from their website. But I’m interested in what is getting stored on the watch / transmitted to my computer. Any news?

  • http://profile.typepad.com/6p017d3d7cbf3f970c Bob T

    Hi Sam! Getting the raw data off of the Basis device requires a good deal more effort (beyond my technical expertise), because you essentially need to intercept (and decrypt) the binary data before it is uploaded to the Basis servers.
    However, there is another project in the works you might want to check out that is more along the lines of what you are looking for – https://github.com/beaugunderson/re-basis

  • http://profile.typepad.com/6p017d3d7cbf3f970c Bob T

    Great, thanks for sharing! I haven’t had much time to investigate the new sets of activity data being collected, but if you are getting a 401 error via python, perhaps you aren’t passing your proper userid in the request you are sending to the server?

  • Chuck Carpenter

    It does look like now there is OAuth authentication. There’s an access token cookie set and you can’t hit the API in a browser without that info. So, login and the path works. If you’re logged out and you get no JSON response. Anyone seen that as well and attempted to authenticate?

  • kevin

    I’ve have seen the same issue. When I type the url (the one created by the scripts) into a browser it works but when it goes through python or the php file in the browser, I run into a 401 error. I am not sure how to setup the credentials within python or the php scripts.
    Thanks.

  • kevin

    Got it to work, I just used the mechanize module in python which allows you to log in and then retrieve the page. The are other approaches using urllib but I found this one to be straight forwrad.
    http://wwwsearch.sourceforge.net/mechanize/

  • http://profile.typepad.com/6p017d3d7cbf3f970c Bob T

    Thanks, Kevin. If it’s a 401 error that could be a basic auth error (just requires username/password) vs. actual Oauth. I’ll try to carve out some time this week to update my script!

  • https://github.com/jonnybones/pybasis JonnyBones

    Nice work Kevin! I’ll look through the mechanize stuff and try to update my scripts if I can.

  • Steve

    I am sorry to be a noob with this, but I downloaded MAMP (as I have experience with it from running Moodle in the past) and I believe I was able to find my Basis ID by looking in me.json when looking at the network tab next to “id:”. I put that string into the appropriate place in the PHP file, and when I load that file in the browser there is no data loading at all (and there is constant data from this moment going back several weeks) though all the headings, etc. load properly:
    https://www.dropbox.com/s/l480dz51jhu1eeo/Screenshot%202014-01-20%2007.45.26.png
    I am sorry to bug you with my missteps, any help would be appreciated.
    Thanks!

  • Nurekasoft

    Hi,
    Thanks for your nice script. However when I try to run your script, I also get the authorization error. My guess is that the access tokens are being set when I log in the browser through cookies and are retrieved for each call by the server. When I clear the cookie, I’m forced to re-login to view the data.
    Please let me know if you were able to update your script to take care of authorization and get it to work.
    Thanks and Regards,
    Nurekasoft

  • http://www.quantifiedbob.com/ Bob

    Just an update on my script – it does indeed appear that Basis has added some basic auth to their API so I will be adding the ability to include username/pass parameters. For security, I’m considering having the user input this at the command line so the data isn’t hard-coded into the script. In the meantime, you can also check out a few of the other scripts (pybasis and rebasis) linked in the comments below!

  • Gustlgast

    There is no user Id on that page anymore. They changed it. No idea how to find that now…

  • Paul VanRoosendaal

    Is bluetooth supported now?

    • http://www.quantifiedbob.com/ Bob

      Hi Paul! Yes, they have enabled the device’s Bluetooth functionality so you can now pair it with your smartphone and wirelessly sync/upload your data using your iOS/Android device.

      • Paul VanRoosendaal

        Thanks Bob. Is there an API yet?

        • http://www.quantifiedbob.com/ Bob

          Unfortunately, no (but their website says they plan to). Also, they added an additional auth layer to their current (non-public) API so my current script is failing until I add in the authentication logic.

  • Pingback: Watching 24 Hours Of Someone's Heartbeat Is Weirdly Mesmerising | Gizmodo Australia

  • RichK

    I just release an app for downloading your basis band data from the basis website. It’s called Basis Retriever and it’s available at http://richk.net/BasisRetriever . It runs on windows, OS X, and should on Linux too, and contains a GUI and command line executables. Hope this helps.

    • http://www.quantifiedbob.com/ Bob

      Thanks, Rich! Looks really good.

      • RichK

        Thanks Bob. I also created a data browser/viewer for data downloaded with the BasisRetriever app and thought I’d share that with the community as well. http://richk.net/wiki/BasisViewer . It’s also on sourceforge as basisviewer.

  • http://www.quantifiedbob.com/ Bob

    Just an update that I have updated the script to work with the recent changes Basis made – you can grab it from Github at https://github.com/btroia/basis-data-export

  • Gustlgast

    Great. Will not run on a Mac.

    • http://www.quantifiedbob.com/ Bob

      Hi, Gustlgast! Actually, I am running it on a Mac. Agree about the include path. Someone else requested including a startdate and enddate so data can be retrieved for longer date ranges by looping through each day (their API times out if you try to request a long date range in a single request). You are more than welcome to fork my code on GitHub and submit a pull request if you have made any improvements!

  • Gustlgast

    Actually it will if I comment out all the fields containing BASIS_*, but when I do so, the script will only run in interactive mode.

  • Gustlgast

    OK let’s be more qualified. It looks like the error I get (It’s actually a weird one, something in hebraeic :)) is due to the fact that the variables are not programmed cleanly. I do not know what kind of php you are using, Bob, but it must be something weird, since I tried on OSX and Linux.

    What I did is correct the variables containing BASIS* like this:

    $options = getopt(“h::u::p::d::f::”);
    $settings = array();
    $settings['basis_username'] = (empty($BASIS_USERNAME)) ? ” : BASIS_USERNAME;
    $settings['basis_password'] = (empty($BASIS_PASSWORD)) ? ” : BASIS_PASSWORD;
    $settings['basis_export_date'] = date(‘Y-m-d’, strtotime(‘now’, time()));

    See they now have a $ in between. The script is running now, but not with the stored user data (in settings). Additionally, at the very beginning, you do require the file BasisExport.class.php.

    Instead of using ./ you should give that a full qualified path, because otherwise calling the script from somewhere else than its own dir will fail.

    So now I can get it to run using cron and the -usomeone@someone.com -pmypassword and so on options.

    Still, good job :)

    G

    • http://www.quantifiedbob.com/ Bob

      I am using PHP 5.5.11 on Mac OS X 10.9.2 (Mavericks). I fixed the require_once() path so if you do a git pull you should be able to run the script from any directory.

  • vg

    Hey Bob,

    would it be possilbe to get the raw data from the basis using the work from the re-basis project combined with your project? Or somehow reverse-engineer the android apk by decompiling it to intercept the data coming over bluetooth to figure out how the basis sends the data and how the phone receives and processes it?