Update on py-openzwave

One of the driving factors behind my plunge into Home Automation is energy conservation.  I believe that usage monitoring is a critical part of conservation; to that end I purchased the excellent ECM-1240 home energy monitor from Brultech.  It’s a nicely integrated serial device with support for monitoring seven independent channels, including pulse devices such as water/gas meters.  While the ECM-1240 is excellent, I was unable to get usage data at a finer level of granularity than the circuit.  I picked up a couple of devices to see which was the best fit for my needs:

The Watts Up Pro device is essentially a more advanced version of the Kill-a-Watt with the addition of a USB port and extensive data logging capabilities.  The Kill-a-Watt was interesting because I wanted to build a Tweet-a-Watt which would wirelessly send usage information via an embedded XBee.  Finally, the Aeon Labs Smart Switch caught my eye as it provides energy usage information as well as the ability to remotely control the connected load.

I did some googling to find as much information as I could about Z-Wave device control.  There are lots of code fragments floating about, but very few turn-key projects that would get me up and running quickly.  Several open-source home automation projects offer Z-Wave support, but I found most of the code to be poorly written, tightly coupled to specific HA projects, or too fragmentary to be of much use.  Finally I stumbled upon the open-zwave project.

The open-zwave project is a GPL-licensed, reverse engineered implementation for the control and management of Z-Wave devices which is unencumbered by Zensys’ licensing.  As I’ve noted before, the “official” Zensys developer kit costs a staggering $1,500 USD, placing it firmly out of reach of the hobbyist.  The open-zwave project is actively developed and maintained and has a very helpful developer community.

One of my goals in developing my own home automation solution is to become proficient in Python.  In the past I have found converting an existing implementation to a new language to be a very effective learning tool, so I rolled up my sleeves and began to port open-zwave to Python.  When I had ported about 50% of the functionality I stumbled across Maarten Damen’s py-openzwave project.  Maarten had taken a different approach; rather than porting the entire library to Python, he used Cython to wrap the existing open-zwave implementation.

Because the open-zwave project is being actively developed, the ugly reality of keeping my Python implementation in sync with the baseline code became more apparent.  I decided to shift gears and run with Maarten’s approach instead.  I forked Maarten’s project on github and got busy coding.  Maarten had ported a couple of methods, but I wanted more extensive support; I spent a couple of days building out the ZWave Manager implementation, adding PyDoc documentation strings, and generally learning how the open-zwave project (and Z-Wave in particular) worked.

I also wrote a Python wrapper class to simplify interaction with the open-zwave implementation, providing a simpler interface which takes care of the low-level details of working with the underlying library.  Some of the interactions with open-zwave are extremely fine-grained.

Curses interface for py-openzwave

Open-zwave library initialization can be an extremely time consuming process, especially in large installations or where there are sleeping devices; Startup takes over a minute on my network of 20 or so Z-Wave devices.  Because of this fact, a simple standalone command line application for device control is impractical.  I am currently developing a set of tools for Z-Wave device management.  The first is a web application which will stay resident and will provide a browser-based GUI as well as a full REST implementation; the second is a curses (text-based) GUI which can either work standalone or use the REST interface back-end.  You can see what the curses interface looks like on this post on Maarten Damen’s blog.

If anyone is interested in contributing to this effort, please feel free to fork my project on GitHub.  My hope is that this project will eventually be incorporated into the open-zwave project itself.

I will be posting more about py-openzwave, ZWave, and my home automation project as I have time to do so.  Please feel free to check out the code, and comments and feedback are extremely welcome!

12 thoughts on “Update on py-openzwave”

  1. Hi,

    I have just downloaded and taken a look at the py-openzwave stuff. Nice work. I am after a simple interface to my z-wave network that I can hack with to do some basic stuff from a tablet PC etc. This looks like it could be a go-er. My z-wave network is tiny, 2 nodes and the controller at the moment and purely for investigating the feasibility at the moment.

    Sorry, I wasn’t sure where was best to add ‘log a ticket’ but I seem to be having a problem with the zwaveCommander example. I seem to be getting one of two different results; either is halts at about 33% of the initial progress or I get to 67% at which point is seems to have gotten the nodes and the values etc, but starts printing out ‘Exception ValueError: “invalid literal for int() with base 10: ”” in ‘openzwave.callback’ ignored’ and doesn’t progress further.

    Can you tell me a) where I should log this issue and b) any ideas how to get around it.

    Many thanks,
    David

    1. Cool, thank you for checking out the project. Keep in mind that py-openzwave is at the mercy of the openzwave project, as we simply wrap their implementation. Openzwave can be *really* slow to initialize – the pauses that you refer to are normal for certain classes of devices. Devices often announce that they support certain features, but never respond when queried for specifics. Other devices are simply buggy. Other things that can cause issues are sleeping devices, moving devices around after you add them to the network, low batteries, etc.

      The first thing to determine is what is causing the delays. The simplest way to test this is to run the test.py application and watch the debug output – it’s quite verbose, but shows what is happening behind the scenes. You will need to run the example under ipython to get any useful results, otherwise it will simply exit immediately and you won’t see much in the way of output. My guess is that the library is timing out waiting for a response to a query; once we find this I’ll show you how to override the behavior and report the issue to the openzwave team.

      We don’t really do issue logging yet – there are simply too many variables with this stuff to get any decent results, and time is hard to come by. Once the project is a bit more complete we’ll be merging with openzwave proper and issues can be tackled by a larger team. Until then, you’re welcome to fork the project on github and hack on it – the help would be much appreciated 🙂

      As it currently sits, zwaveCommander is more of an exploratory tool than a practical utility. The intent of ZWaveServer is to provide a RESTful interface to openzwave, it can be left running so that the startup/initialization time isn’t hit every time it’s executed. Once the server is complete, zwaveCommander will be updated to support using the server as a back-end, speeding up initialization.

      Can you let me know what devices you are using? What controller? Operating system and version?

      Thanks again for your interest!

  2. Hi Steve,

    Thanks for getting back to me.

    I am using all Aeon Labs stuff. Here in Australia, we only have one supplier of compatible z-wave gear, which is shown here http://www.smarthome.com.au/zseries/z-wave-products.php . I have the USB Controller, In-Wall Appliance Controller and Lamp Controller. I am testing with the Ubuntu Lucid (10.04) OS.

    When I run ipython test.py, I get a lot of output that seems to scroll past but no logs of errors that I can see. It seems to identify all three nodes, the current levels, the power usage of the 2 nodes, whether they are included in the all-on/all-off scene etc, etc.

    Using github is new to me, so I will have to get my head around how that works as a versioning system too. I am also going to have to dig into how the zwaveServer and zwaveCommander are interacting with the manager. The test.py program is making more sense to me now I have fully ready those few lines of code and clearly ready the output. The event drive model provided by the ‘watcher’ or the callback is exactly what I have been hoping to do. Glad I am not re-inventing any wheels.

    I see there are four branches/forks of this project on github. I am currently running maartendamen’s, is there a better one to be running? Also, what I downloaded was a snapshot of his repository. I guess I should really be doing this differently to make sure I always have access to the latest checkins.

    Anyways, getting a little late here so have to get off. Thanks again,

    Regards,
    David

  3. I really like the Aeon Labs devices; unfortunately the “bespoke” in-wall devices aren’t yet available in the states. I’ve got a couple of the smart switches (which appear similar to the .au “Appliance Controller”), they were *really* slow to start up until I disabled one of the value queries via openzwave configuration.

    My fork (buzzdavidson) contains the latest “bleeding edge” code – Maarten has given me commit rights to his version but I have yet to transition over (free time being what it is). You may want to switch to my fork to see if you have better luck.

    Most of the work that I’ve done is in the ZWaveWrapper class (contained in examples/common/ozwWrapper.py); this class handles callbacks and all of the fiddly little details that have to be managed when using openzwave. Both the server and commander examples use this class behind the scenes. I’d *highly* recommend using this class rather than interfacing with Manager directly, as Manager is kind of a PITA to use directly.

    AFAIK the only truly active forks are mine and Maarten D’s, and I don’t think Maarten has touched the code for a while. Drew P did some really important preliminary work (introducing the existing callback mechanism), but has been notably silent since then. A shame, because he had some really valuable insights to share. Such is the way with social coding!

    In any case, the long startup times are probably due to some additional queries that the Wrapper is performing that are not called via test.py; I wouldn’t be surprised to find that the in-wall controllers share some of the smart switch’s quirky behaviors. The other issue is probably related to a fairly recent merge of Drew’s cython updates (in openzwave.pyx). The implementation of the setValue method is really kludged together at the moment, an indication of my weak C++ skills :P. Feel free to take a look at the code and poke fun at the implementation, it really needs some help!

  4. Hmmm, so I managed to fudge it a little to write all of the ozw logging to a file. It seems that node3 is timing out when requested for a value, just as you suggested.

    It seems to have three attempts, get no response three times and it just so happens that the ‘invalid literal for int()’ message is output three times. Perhaps a coincidence or perhaps not.

    The following log shows a valid command and response for SensorMultilevelCmd_Get and the failing command for MeterCmd_Get.

    2011-06-21 21:23:00:816 Sending command (Callback ID=0x24, Expected Reply=0x04) – SensorMultilevelCmd_Get (Node=3): 0x01, 0x09, 0x00, 0x13, 0x03, 0x02, 0x31, 0x04, 0x05, 0x24, 0xf0
    2011-06-21 21:23:00:821 ACK received CallbackId 0x24 Reply 0x04
    2011-06-21 21:23:00:822 Received: 0x01, 0x04, 0x01, 0x13, 0x01, 0xe8
    2011-06-21 21:23:00:822 ZW_SEND_DATA delivered to Z-Wave stack
    2011-06-21 21:23:00:835 Received: 0x01, 0x05, 0x00, 0x13, 0x24, 0x00, 0xcd
    2011-06-21 21:23:00:835 ZW_SEND_DATA Request with callback ID 0x24 received (expected 0x24)
    2011-06-21 21:23:00:843 Received: 0x01, 0x0e, 0x00, 0x04, 0x00, 0x03, 0x08, 0x31, 0x05, 0x04, 0x64, 0x00, 0x00, 0x00, 0x00, 0xaa

    2011-06-21 21:23:00:843 Received SensorMultiLevel report from node 3, instance 1: value=0.000W
    2011-06-21 21:23:00:843 Expected reply and command class was received
    2011-06-21 21:23:00:843 Message transaction complete

    2011-06-21 21:23:00:844 Sending command (Callback ID=0x25, Expected Reply=0x04) – MeterCmd_Get (Node=3): 0x01, 0x09, 0x00, 0x13, 0x03, 0x02, 0x32, 0x01, 0x05, 0x25, 0xf7
    2011-06-21 21:23:00:849 ACK received CallbackId 0x25 Reply 0x04
    2011-06-21 21:23:00:850 Received: 0x01, 0x04, 0x01, 0x13, 0x01, 0xe8
    2011-06-21 21:23:00:850 ZW_SEND_DATA delivered to Z-Wave stack
    2011-06-21 21:23:00:863 Received: 0x01, 0x05, 0x00, 0x13, 0x25, 0x00, 0xcc
    2011-06-21 21:23:00:863 ZW_SEND_DATA Request with callback ID 0x25 received (expected 0x25)
    2011-06-21 21:23:06:067 Timeout
    2011-06-21 21:23:06:067 Resending message (attempt 1)

    2011-06-21 21:23:06:167 Sending command (Callback ID=0x25, Expected Reply=0x04) – MeterCmd_Get (Node=3): 0x01, 0x09, 0x00, 0x13, 0x03, 0x02, 0x32, 0x01, 0x05, 0x25, 0xf7
    2011-06-21 21:23:06:173 ACK received CallbackId 0x25 Reply 0x04
    2011-06-21 21:23:06:173 Received: 0x01, 0x04, 0x01, 0x13, 0x01, 0xe8
    2011-06-21 21:23:06:173 ZW_SEND_DATA delivered to Z-Wave stack
    2011-06-21 21:23:06:187 Received: 0x01, 0x05, 0x00, 0x13, 0x25, 0x00, 0xcc
    2011-06-21 21:23:06:187 ZW_SEND_DATA Request with callback ID 0x25 received (expected 0x25)
    2011-06-21 21:23:11:387 Timeout
    2011-06-21 21:23:11:387 Resending message (attempt 2)

    2011-06-21 21:23:11:487 Sending command (Callback ID=0x25, Expected Reply=0x04) – MeterCmd_Get (Node=3): 0x01, 0x09, 0x00, 0x13, 0x03, 0x02, 0x32, 0x01, 0x05, 0x25, 0xf7
    2011-06-21 21:23:11:491 ACK received CallbackId 0x25 Reply 0x04
    2011-06-21 21:23:11:496 Received: 0x01, 0x04, 0x01, 0x13, 0x01, 0xe8
    2011-06-21 21:23:11:496 ZW_SEND_DATA delivered to Z-Wave stack
    2011-06-21 21:23:11:507 Received: 0x01, 0x05, 0x00, 0x13, 0x25, 0x00, 0xcc
    2011-06-21 21:23:11:507 ZW_SEND_DATA Request with callback ID 0x25 received (expected 0x25)
    2011-06-21 21:23:16:707 ERROR: Dropping command, expected response not received after three attempts

    (Sorry pasting so much log)

    So, I need to prevent the MeterCmd_Get request from being made. I guess this is done in some way via the files in openzwave/config/ but I will have to dig deeper into that now. Feel free to prod me in the right direction though… 😛

    Cheers,
    David

  5. Although (just to confuse things a little more) the command seems to work sometimes, yet I get the same invalid literal errors.

    2011-06-21 22:32:05:764 Sending command (Callback ID=0x25, Expected Reply=0x04) – MeterCmd_Get (Node=3): 0x01, 0x09, 0x00, 0x13, 0x03, 0x02, 0x32, 0x01, 0x05, 0x25, 0xf7
    2011-06-21 22:32:05:769 ACK received CallbackId 0x25 Reply 0x04
    2011-06-21 22:32:05:769 Received: 0x01, 0x04, 0x01, 0x13, 0x01, 0xe8
    2011-06-21 22:32:05:769 ZW_SEND_DATA delivered to Z-Wave stack
    2011-06-21 22:32:05:783 Received: 0x01, 0x05, 0x00, 0x13, 0x25, 0x00, 0xcc
    2011-06-21 22:32:05:783 ZW_SEND_DATA Request with callback ID 0x25 received (expected 0x25)
    2011-06-21 22:32:05:795 Received: 0x01, 0x14, 0x00, 0x04, 0x00, 0x03, 0x0e, 0x32, 0x02, 0x21, 0x64, 0x00, 0x00, 0x05, 0x72, 0x00, 0x00, 0x00, 0x00, 0x05, 0x72, 0x97

    2011-06-21 22:32:05:795 Received Meter report from node 3: value=1.394
    2011-06-21 22:32:05:795 Expected reply and command class was received
    2011-06-21 22:32:05:795 Message transaction complete

    Doh!

  6. Hey Steve,

    Sorry, I really should just leave you alone now, although I was about to give up and go to bed but I think I have found the problem. 😛

    I added some debug to the ozwWrapper and have found that the problem is caused in the _updateNodeNeighbors method of ozwWrapper.py.

    Sometimes, the received neighborstr is, for example ‘(1,)’, which when stripped of parenthesis and split out renders [‘1’,”] and when we iterate through the list, we try and cast the empty string to an int, which throws the exception.

    I added a filter to the conversion of neighborstr to a list which removes the empty strings as follows…

    node._neighbors = sorted([int(i) for i in filter(None, neighborstr.strip(‘()’).split(‘,’))])

    …and it all seems to be humming along now (although I can only make one change to my dimmable nodes per each execution of the zwaveCommander program but that is tomorrows task 🙂

    Good night,
    David

  7. Wow, great find. Your feedback is most welcome – the more the merrier! Keep the comments coming!

    First, about the COMMAND_CLASS_SENSOR_MULTILEVEL timeout; this is the exact same behavior that I get with my Aeon Labs Smart Switch. The fix is to tell openzwave not to query for that particular value (the same information is available via COMMAND_CLASS_METER. More details in this thread. Adding that configuration entry sped up initialization considerably.

    OpenZWave exhibits some really strange behaviors with the neighbor node stuff, if you look at the wrapper code you’ll see several corner cases already handled (the most peculiar being that sleeping nodes return long, seemingly random lists of neighbors). I haven’t seen the specific behavior that you mention – the value ‘(1,)’ is a tuple with a single value (a oneple?), there is probably a more efficient Python mechanism for handling this than string manipulation. This is my first “big” Python project, so I’m still learning.

    The single dimming step is another quirk of OpenZWave, IMHO, at least in the way that I am using it. It’s a known issue. Here’s what’s happening: at startup, based upon the command classes supported by the specific device, OpenZWave maps COMMAND_CLASS_BASIC to the most appropriate class on the device. This is done behind the scenes in the OpenZWave code. COMMAND_CLASS_BASIC is a nice idea, basically allowing simple controllers to send simple commands to a device (on/off/dim/bright) without knowing or caring about the device specifics. In this respect, it’s kind of similar to how X10 operates. The problem in this particular case is that there is no feedback mechanism to the original mapped command class; zwaveCommander sends a COMMAND_CLASS_BASIC message to the device, but the mapped command class and associated value nodes are never notified via a callback. The problem here is maintaining a generic solution. OpenZWave does not provide access to the specific COMMAND_CLASS_BASIC mapping – it is buried within private members in classes in the C++ code. The fix is to not use the manager’s setLevel(), setOn(), and setOff() commands, but rather to use the setValue methods instead – I just haven’t gotten to it yet. I imagine I should submit a bug to the OZW team about this one.

    Anyway, keep up the good work – I look forward to seeing what you do with the project. Check out the zwaveServer code if you get a chance as well, I’d be interested in your feedback (as a shortcut, you need to access the page before the library initializes. It’s on the TODO list.) My free time is currently devoted to a little Arduino sensor project, but I’ll be getting back to py-openzwave in the next week or two.

    FWIW, my house is about 50% Insteon and 50% ZWave, and honestly there are things that I like about both solutions. The Insteon stuff is solidly built, looks nice, and is cheaper than the Leviton ZWave equipment that I’m using, but people have complaints about reliability.

  8. Hi Steve – just wondering what’s going on with py-openzwave at this point. Your fork at github doesn’t show much activity – just wondering if you’re still active with it or not. Thanks!

    1. my changes were merged back into Maarten Damen’s trunk version a while ago. I haven’t been active in the project since last summer, but will probably dive back in this spring.

  9. Hello Steve,
    I’ve been playing with py-openzwave for the past 2 weeks. Everything looked great until I discovered(more like got stock) that there’s no REMOVE NODE function. I’m sure I’ll figure it out eventually, but I guess I’m too lazy, so I was hoping I could get an advice from someone experienced in this area. I’m using Aeon ZStick S2, it supports hardware node add/remove by pressing a button on USB controller, but I’m looking at programmable way of doing it(I did mention that I’m lazy, right? Don’t like an idea of wondering around the house and doing extra clicks:) ) Any suggestions regarding the subject matter?

    Thanks,
    Andrei

    1. Andrei,

      py-openzwave hasn’t gotten much attentention lately, glad to hear that you’re playing with it. Since the application really hasn’t been worked on in the last year it is probably desperately in need of a refresh and synchronization with the main openzwave project.

      I’ve always performed node removal using the native hardware support rather than programmatically; as far as I know that capability doesn’t even exist. I don’t often remove devices. I’d suggest checking with the main openzwave project to see if this is a supported capability.

Leave a Reply

Your email address will not be published. Required fields are marked *