3rd Party Local API - Beta Testers/Contributers Wanted

[donnyk]

Hi All,

I received and installed my envisalink 3 a few weeks ago and given the lack of a real functional local (or remote for that matter) easy to use API I've been working on one off and on since then.

The feature list so far:
-Written in python (runnable on almost any os as well as modded/open source routers running ddwrt/tomato)
-Connects and maintains a connection to your envisalink device
-Maintains a state of all the alarm zones and partitions
-Runs a very simple http server
-Currently very simple (because I suck and hate web design) front page to arm/stayarm/disarm alarm.
-JSON output of the status of entire system, including all zones and partitions.
-Sends notifications using pageonce.

Todo list:
-Add some form of authentication
-Add email, and other notifications.
-Add zone and partition names
-Finish a web page with alarm keypad and other action buttons buttons, zone/partition status using a timed/repeating AJAX/Xhtmlrequest to pull data from the json api.
-Other requests?

I'm like to share the code with a few willing participants if anyone is interested. When I get it to a more complete stage I plan to post the source on github. As well if anyone has any feature requests etc, i'd be interested to hear them.

Thanks!

Here is some sample output of my system pulled via http://host/api :

Code: Select all

{"zones": {"1": {"status": {"open": false}, "lastevents": [{"message": "Zone 001 Restored", "datetime": "2012-12-02 17:21:06"}, {"message": "Zone 001 Open", "datetime": "2012-12-02 18:40:45"}, {"message": "Zone 001 Restored", "datetime": "2012-12-02 18:40:51"}]}, "2": {"status": {"open": false}, "lastevents": [{"message": "Zone 002 Restored", "datetime": "2012-12-02 17:21:06"}, {"message": "Zone 002 Open", "datetime": "2012-12-02 17:55:17"}, {"message": "Zone 002 Restored", "datetime": "2012-12-02 17:55:24"}, {"message": "Zone 002 Open", "datetime": "2012-12-02 18:17:17"}, {"message": "Zone 002 Restored", "datetime": "2012-12-02 18:17:20"}, {"message": "Zone 002 Open", "datetime": "2012-12-02 18:40:33"}, {"message": "Zone 002 Restored", "datetime": "2012-12-02 18:40:37"}, {"message": "Zone 002 Open", "datetime": "2012-12-02 18:44:53"}, {"message": "Zone 002 Restored", "datetime": "2012-12-02 18:45:01"}, {"message": "Zone 002 Open", "datetime": "2012-12-02 23:31:34"}, {"message": "Zone 002 Restored", "datetime": "2012-12-02 23:33:01"}]}, "3": {"status": {"open": false}, "lastevents": [{"message": "Zone 003 Restored", "datetime": "2012-12-02 17:21:06"}]}, "4": {"status": {"open": false}, "lastevents": [{"message": "Zone 004 Restored", "datetime": "2012-12-02 17:21:06"}]}, "5": {"status": {"open": false}, "lastevents": [{"message": "Zone 005 Restored", "datetime": "2012-12-02 17:21:06"}, {"message": "Zone 005 Open", "datetime": "2012-12-02 17:22:43"}, {"message": "Zone 005 Restored", "datetime": "2012-12-02 17:22:46"}, {"message": "Zone 005 Open", "datetime": "2012-12-02 17:40:58"}, {"message": "Zone 005 Restored", "datetime": "2012-12-02 17:40:59"}, {"message": "Zone 005 Open", "datetime": "2012-12-02 17:41:00"}, {"message": "Zone 005 Restored", "datetime": "2012-12-02 17:41:02"}, {"message": "Zone 005 Open", "datetime": "2012-12-02 17:42:14"}, {"message": "Zone 005 Restored", "datetime": "2012-12-02 17:42:15"}, {"message": "Zone 005 Open", "datetime": "2012-12-02 17:53:56"}, {"message": "Zone 005 Restored", "datetime": "2012-12-02 23:37:07"}]}, "6": {"status": {"open": false}, "lastevents": [{"message": "Zone 006 Restored", "datetime": "2012-12-02 17:21:06"}]}, "7": {"status": {"open": false}, "lastevents": [{"message": "Zone 007 Restored", "datetime": "2012-12-02 17:21:06"}]}, "8": {"status": {"open": false}, "lastevents": [{"message": "Zone 008 Restored", "datetime": "2012-12-02 17:21:06"}]}, "9": {"status": {"open": false}, "lastevents": [{"message": "Zone 009 Restored", "datetime": "2012-12-02 17:21:06"}]}, "10": {"status": {"open": false}, "lastevents": [{"message": "Zone 010 Restored", "datetime": "2012-12-02 17:21:06"}]}, "11": {"status": {"open": false}, "lastevents": [{"message": "Zone 011 Restored", "datetime": "2012-12-02 17:21:07"}]}, "12": {"status": {"open": false}, "lastevents": [{"message": "Zone 012 Restored", "datetime": "2012-12-02 17:21:07"}]}, "13": {"status": {"open": false}, "lastevents": [{"message": "Zone 013 Restored", "datetime": "2012-12-02 17:21:07"}]}, "14": {"status": {"open": false}, "lastevents": [{"message": "Zone 014 Restored", "datetime": "2012-12-02 17:21:07"}]}, "15": {"status": {"open": true}, "lastevents": [{"message": "Zone 015 Open", "datetime": "2012-12-02 17:21:07"}]}, "16": {"status": {"open": false}, "lastevents": [{"message": "Zone 016 Restored", "datetime": "2012-12-02 17:21:07"}]}, "17": {"status": {"open": false}, "lastevents": [{"message": "Zone 017 Restored", "datetime": "2012-12-02 17:21:07"}]}, "18": {"status": {"open": false}, "lastevents": [{"message": "Zone 018 Restored", "datetime": "2012-12-02 17:21:07"}]}, "19": {"status": {"open": false}, "lastevents": [{"message": "Zone 019 Restored", "datetime": "2012-12-02 17:21:07"}]}, "20": {"status": {"open": false}, "lastevents": [{"message": "Zone 020 Restored", "datetime": "2012-12-02 17:21:07"}]}, "21": {"status": {"open": false}, "lastevents": [{"message": "Zone 021 Restored", "datetime": "2012-12-02 17:21:07"}]}, "22": {"status": {"open": false}, "lastevents": [{"message": "Zone 022 Restored", "datetime": "2012-12-02 17:21:08"}]}, "23": {"status": {"open": false}, "lastevents": [{"message": "Zone 023 Restored", "datetime": "2012-12-02 17:21:08"}]}, "24": {"status": {"open": false}, "lastevents": [{"message": "Zone 024 Restored", "datetime": "2012-12-02 17:21:08"}]}, "25": {"status": {"open": false}, "lastevents": [{"message": "Zone 025 Restored", "datetime": "2012-12-02 17:21:08"}]}, "26": {"status": {"open": false}, "lastevents": [{"message": "Zone 026 Restored", "datetime": "2012-12-02 17:21:08"}]}, "27": {"status": {"open": false}, "lastevents": [{"message": "Zone 027 Restored", "datetime": "2012-12-02 17:21:08"}]}, "28": {"status": {"open": false}, "lastevents": [{"message": "Zone 028 Restored", "datetime": "2012-12-02 17:21:08"}]}, "29": {"status": {"open": false}, "lastevents": [{"message": "Zone 029 Restored", "datetime": "2012-12-02 17:21:08"}]}, "30": {"status": {"open": false}, "lastevents": [{"message": "Zone 030 Restored", "datetime": "2012-12-02 17:21:08"}]}, "31": {"status": {"open": false}, "lastevents": [{"message": "Zone 031 Restored", "datetime": "2012-12-02 17:21:08"}]}, "32": {"status": {"open": false}, "lastevents": [{"message": "Zone 032 Restored", "datetime": "2012-12-02 17:21:08"}]}, "33": {"status": {"open": false}, "lastevents": [{"message": "Zone 033 Restored", "datetime": "2012-12-02 17:21:08"}]}, "34": {"status": {"open": false}, "lastevents": [{"message": "Zone 034 Restored", "datetime": "2012-12-02 17:21:08"}]}, "35": {"status": {"open": false}, "lastevents": [{"message": "Zone 035 Restored", "datetime": "2012-12-02 17:21:08"}]}, "36": {"status": {"open": false}, "lastevents": [{"message": "Zone 036 Restored", "datetime": "2012-12-02 17:21:08"}]}, "37": {"status": {"open": false}, "lastevents": [{"message": "Zone 037 Restored", "datetime": "2012-12-02 17:21:08"}]}, "38": {"status": {"open": false}, "lastevents": [{"message": "Zone 038 Restored", "datetime": "2012-12-02 17:21:08"}]}, "39": {"status": {"open": false}, "lastevents": [{"message": "Zone 039 Restored", "datetime": "2012-12-02 17:21:08"}]}, "40": {"status": {"open": false}, "lastevents": [{"message": "Zone 040 Restored", "datetime": "2012-12-02 17:21:08"}]}, "41": {"status": {"open": false}, "lastevents": [{"message": "Zone 041 Restored", "datetime": "2012-12-02 17:21:08"}]}, "42": {"status": {"open": false}, "lastevents": [{"message": "Zone 042 Restored", "datetime": "2012-12-02 17:21:08"}]}, "43": {"status": {"open": false}, "lastevents": [{"message": "Zone 043 Restored", "datetime": "2012-12-02 17:21:08"}]}, "44": {"status": {"open": false}, "lastevents": [{"message": "Zone 044 Restored", "datetime": "2012-12-02 17:21:09"}]}, "45": {"status": {"open": false}, "lastevents": [{"message": "Zone 045 Restored", "datetime": "2012-12-02 17:21:09"}]}, "46": {"status": {"open": false}, "lastevents": [{"message": "Zone 046 Restored", "datetime": "2012-12-02 17:21:09"}]}, "47": {"status": {"open": false}, "lastevents": [{"message": "Zone 047 Restored", "datetime": "2012-12-02 17:21:09"}]}, "48": {"status": {"open": false}, "lastevents": [{"message": "Zone 048 Restored", "datetime": "2012-12-02 17:21:09"}]}, "49": {"status": {"open": false}, "lastevents": [{"message": "Zone 049 Restored", "datetime": "2012-12-02 17:21:09"}]}, "50": {"status": {"open": false}, "lastevents": [{"message": "Zone 050 Restored", "datetime": "2012-12-02 17:21:09"}]}, "51": {"status": {"open": false}, "lastevents": [{"message": "Zone 051 Restored", "datetime": "2012-12-02 17:21:09"}]}, "52": {"status": {"open": false}, "lastevents": [{"message": "Zone 052 Restored", "datetime": "2012-12-02 17:21:09"}]}, "53": {"status": {"open": false}, "lastevents": [{"message": "Zone 053 Restored", "datetime": "2012-12-02 17:21:09"}]}, "54": {"status": {"open": false}, "lastevents": [{"message": "Zone 054 Restored", "datetime": "2012-12-02 17:21:09"}]}, "55": {"status": {"open": false}, "lastevents": [{"message": "Zone 055 Restored", "datetime": "2012-12-02 17:21:09"}]}, "56": {"status": {"open": false}, "lastevents": [{"message": "Zone 056 Restored", "datetime": "2012-12-02 17:21:09"}]}, "57": {"status": {"open": false}, "lastevents": [{"message": "Zone 057 Restored", "datetime": "2012-12-02 17:21:09"}]}, "58": {"status": {"open": false}, "lastevents": [{"message": "Zone 058 Restored", "datetime": "2012-12-02 17:21:09"}]}, "59": {"status": {"open": false}, "lastevents": [{"message": "Zone 059 Restored", "datetime": "2012-12-02 17:21:09"}]}, "60": {"status": {"open": false}, "lastevents": [{"message": "Zone 060 Restored", "datetime": "2012-12-02 17:21:09"}]}, "61": {"status": {"open": false}, "lastevents": [{"message": "Zone 061 Restored", "datetime": "2012-12-02 17:21:09"}]}, "62": {"status": {"open": false}, "lastevents": [{"message": "Zone 062 Restored", "datetime": "2012-12-02 17:21:09"}]}, "63": {"status": {"open": false}, "lastevents": [{"message": "Zone 063 Restored", "datetime": "2012-12-02 17:21:09"}]}, "64": {"status": {"open": false}, "lastevents": [{"message": "Zone 064 Restored", "datetime": "2012-12-02 17:21:09"}]}}, "version": 0.1, "partition": {"1": {"status": {"ready": false, "trouble": false, "armed": true}, "lastevents": [{"message": "Partition 1 Armed Mode 1", "datetime": "2012-12-02 17:21:09"}, {"message": "Partition 1 Trouble LED OFF", "datetime": "2012-12-02 17:21:10"}, {"message": "Partition 1 User 0040 Opening", "datetime": "2012-12-02 17:55:24"}, {"message": "Partition 1 Disarmed", "datetime": "2012-12-02 17:55:24"}, {"message": "Partition 1 Not Ready", "datetime": "2012-12-02 18:17:17"}, {"message": "Partition 1 Not Ready", "datetime": "2012-12-02 18:40:33"}, {"message": "Partition 1 Not Ready", "datetime": "2012-12-02 18:40:45"}, {"message": "Partition 1 Not Ready", "datetime": "2012-12-02 18:44:53"}, {"message": "Partition 1 Not Ready", "datetime": "2012-12-02 23:31:34"}, {"message": "Partition 1 Not Ready", "datetime": "2012-12-02 23:32:56"}, {"message": "Partition 1 Armed Mode 1", "datetime": "2012-12-03 00:06:50"}]}, "2": {"status": {"trouble": false}, "lastevents": [{"message": "Partition 2 Trouble LED OFF", "datetime": "2012-12-02 17:21:10"}]}, "3": {"status": {"trouble": false}, "lastevents": [{"message": "Partition 3 Trouble LED OFF", "datetime": "2012-12-02 17:21:10"}]}, "4": {"status": {"trouble": false}, "lastevents": [{"message": "Partition 4 Trouble LED OFF", "datetime": "2012-12-02 17:21:10"}]}, "5": {"status": {"trouble": false}, "lastevents": [{"message": "Partition 5 Trouble LED OFF", "datetime": "2012-12-02 17:21:10"}]}, "6": {"status": {"trouble": false}, "lastevents": [{"message": "Partition 6 Trouble LED OFF", "datetime": "2012-12-02 17:21:10"}]}, "7": {"status": {"trouble": false}, "lastevents": [{"message": "Partition 7 Trouble LED OFF", "datetime": "2012-12-02 17:21:10"}]}, "8": {"status": {"trouble": false}, "lastevents": [{"message": "Partition 8 Trouble LED OFF", "datetime": "2012-12-02 17:21:10"}]}}}

[PurdueGuy]

I would like to try and help, but currently my home automation system uses the TPI for status, reporting, etc. If you could add a pass-through (since the TPI only allows 1 external connection), I could help test while keeping my home automation system in the loop.

[donnyk]

I would like to try and help, but currently my home automation system uses the TPI for status, reporting, etc. If you could add a pass-through (since the TPI only allows 1 external connection), I could help test while keeping my home automation system in the loop.

PurdueGuy,

A passthrough/proxy of sorts is a interesting idea. I did think about that, but chose not to proceed with it as a priority. I wanted to make a local API that was simple to program against and easy to get a state from without having to parse 80+ events. Outputting everything in JSON over http also makes the output very portable and usable/readable on almost any platform.

However, I do see your point, there may be situations where you want to integrate with an existing setup without breaking it. Given the restriction of one connection to the envisalink this would be worth looking into for sure. I will look into this over the next few days. Let me know if you have any other thoughts on it.

[gschrader]

I'd be interested in taking a look. I was considering starting something similar myself (not in python mind you) your feature set pretty much captures what I wanted to do. I might be able to help with some of the web design, I'm more developer than designer myself but I've found lately using Bootstrap makes a pretty decent looking UI.

[donnyk]

I'd be interested in taking a look. I was considering starting something similar myself (not in python mind you) your feature set pretty much captures what I wanted to do. I might be able to help with some of the web design, I'm more developer than designer myself but I've found lately using Bootstrap makes a pretty decent looking UI.

The reason I chose python was the rapid development aspect as well as the portability. Python will run anywhere a python interpreter exists, and that includes my home router which is running an open source firmware called tomato (dd-wrt works as well).

I'm actually playing with two competing versions now. The first one and the one I was working on when I wrote the original post is purely python but uses a python module called Asyncore that many would consider 'frustrating'. It is nice though that to run it requires no addons just a base install of python. To me that is important.

Based on some suggestions however I decided to give Twisted a try. Twisted is a networking stack for python that greatly simplifies things. However it requires the user to install an additional python addon/plugin. Its not hard to do but it is additional steps I would like to avoid if possible. I did not complete this version, but I did port the client aspect over to Twisted. It works but again I am hesitant about requiring additional components for python.

As for the web design I am leaning towards being via minimalist. A tiny CSS/JS library would be fine. I would probably read the files into memory when the script is started and serve them out of memory anyway.

I am also in the process of working on the TPI proxy component that was previously asked for. Once that is done and I iron a few things out I will post the code. Should hopefully be by the end of the weekend.

[gschrader]

yeah I see your reasoning for python, my crappy router can run dd-wrt but I'm not sure it has enough memory or storage for the python package. I'm thinking something like a raspberry pi or something would be just as ideal for this so something lightweight would be beneficial to that as well.

Anyway, I'll have a look once you make it available.

[donnyk]

yeah I see your reasoning for python, my crappy router can run dd-wrt but I'm not sure it has enough memory or storage for the python package. I'm thinking something like a raspberry pi or something would be just as ideal for this so something lightweight would be beneficial to that as well.

Anyway, I'll have a look once you make it available.

Does your router have a USB port?

I'm almost finished the TPI proxy someone else requested, I just need to emulate the TPI authentication other then that it works great.

[donnyk]

I would like to try and help, but currently my home automation system uses the TPI for status, reporting, etc. If you could add a pass-through (since the TPI only allows 1 external connection), I could help test while keeping my home automation system in the loop.

PurdueGuy,

Thanks again for the pass-through/suggestion. I finished it tonight.

As requested you can make multiple TPI connections now to the proxy while the proxy holds one single TPI connection. As mentioned before you can still make http requests and pull information about the current alarm status over http.

One last thing I need to address is if the connection to the TPI port is lost. Once I finish that I'll post the code for you to try.

[pounder]

Keep in mind there is a fair amount of state you need to track when doing the fan out proxy everyone is interested in.

eg:

- you need to keep track of each client logging in
- you need to track the timestamp format they ask for
- you need to track if any client goes into master or installer mode and lock out the others and filter messages accordingly
- you need to send broadcasts to everyone but specific ack/nak only to the client that send the command.

I whipped up a python client as well, so far mine connects to the module and then just dumps everything into postgres and accepts commands. My plan is to use triggers on the raw event table to crack the messages into more meaningful things like zone table updates, etc., and then the port proxy or whatever other interfaces can be driven off the database, rather than the python bridge piece. (data to the module goes the same way, trigger off a table sends the command to the python piece)

Once the data is in a "massaged" form that has state, any number of socket, php, snmp, or other clients can use the same store of data.

[donnyk]

Keep in mind there is a fair amount of state you need to track when doing the fan out proxy everyone is interested in.

eg:

- you need to keep track of each client logging in
- you need to track the timestamp format they ask for
- you need to track if any client goes into master or installer mode and lock out the others and filter messages accordingly
- you need to send broadcasts to everyone but specific ack/nak only to the client that send the command.

Hi Pounder,

-I am already keeping track of each proxy client who logs in.
-I did not think about the timestamp format, what I will do is enable it for the main connection and then proxy it out to any clients who enable it. Anyone who does not I will chop it off before relaying the event. The same method would apply to the time broadcast control.
-I am not sure there is a big use case for going into master/installer mode via this mechanism what do you think. It may be best just to blacklist those commands. I don't see why anyone would want to do any programming this way, is there a reason I am missing?
-My initial plan was not to filter down the ack/nak's to the client who requested them as most/all? clients should hopefully be able to handle all of the events at any time regardless if they requested them or not. If that is not the case I can look into it for sure, it should be doable. The issue would be of course if an event happens while you are requesting something. It would have to be pretty smart to know based on the request sent which type of event would come back.