Thursday, 28 September 2017

Handling the Digital World

At present, I am watching several films, reading mail, and pulling thoughts together on a separate Doc, oh, and drawing in a CAD package. I use several screens to keep this little lot in the air and viable.
My mac is at present pushing a lot of air through its self, keeping up with all of this, not surprising really given the work load I am putting on it.

I work from a small home office, but today I am in the dining room, it takes about 20 min to set up,  or decamp the usual equipment from my small office to the dining room, but I like to keep my Flatcoated Lab happy

Data storage is now easy, Google drive handles it all, except for my CAD files which I still store on Dropbox, a legacy from earlier days and one I will eventually move to drive. but other projects need more time to evaluate, such as my sensor project, howwill I handle the massive data I know will be generated.

For this, I need to gain a better understanding of sensors their generation of data, routing of information and central collection, how I can organise the storage and interpret the data against other data sources.

A simple and easy first attempt to store a small data collection, is to use Google Drive spreadsheet,  my source is a single house, not too big and not so small, I reckon that as a start, I will install as many sensors I can, say 5 and set up a small google spreadsheet, and feed data to it via iFTTT, moving up to around 20 sensors looking at

  • Temp
  • Humidity
  • Sound
  • Light
  • Pressure

at various levels and positions within the house to get the best spread of data.

Each Google Spreadsheet has a limit of 400,000 cells, with a maximum of 256 columns per sheet. There are also other limitations: Number of Formulas: 40,000 cells containing formulas. Number of Tabs: 200 sheets per workbook. So in the near future this is going to fill and I need to have moved on, Google has a neat answer here called cloud.iot.core to handle this type of data on a large scale, and don't forget even 20 sensors is going to fill this in just a few years, I am specifying in my project spec notes that the life of the house is going to be 120 years. add external data to show the calculation complete, then a spreadsheet is not going to last, but it's a good start to understand the data and iron out some simple collection problems ready to move to the bigger system.

Next, what sensor system do I use there are several off the shelf systems that sell units, but they are expensive but an easy fix, next up is the Raspberry pie and Arduino small, and I mean small computers, internally they are very useful, external they need a robust power supply, battery is not really a long term option.

Then how do I collect this data, well the off the shelf units are mostly compatible with iFTTT a usefull interface service that's free and I use a lot to feed information to Evernote, yes I know I don't use it much, but the links are still there.

The later Raspberry pie and Arduino units will operate as wifi linked computers and can be easily linked to a google spreadsheet.

A web spreadsheet model or handling streaming data
A Spreadsheet Tool for Creating WebApplications Using Online Data
A spreadsheet approach to programming & managing Sensor Networks
Save Edyn Garden Sensor data to a Google spreadsheet (Using iFTTT )
Using Google Spreadsheets for Logging Sensor Data
Gathering Data with IFTTT