electronics

Hi there, and welcome to our monthly status update!

TL;DR: We’ve passed FCC certification, and CE is in progress. We’re progressing with assembly, loads of UHK boxes arrived from the printing factory, and the UHK accessory boxes are already packed. The firmware and Agent have matured substantially. We’re waiting for final answers from TÜV to proceed further.

We’re aiming to send out the pilot run UHKs in October, and start the delivery of the rest of the first batch of 2,000 UHKs in November. If you backed us after 2017-07-13 then you’re in the second batch, which is expected to ship in March 2018.

FFC and CE certification

A week ago, we got great news from TÜV:

“All FCC 15 ready and just PASS for Radiated Emission. Producing the report now.”

Then I asked about CE, and to my surprise they told me that they didn’t know that we also needed CE measurements. I searched my emails to see whether I miscommunicated something, but I didn’t. Not only did I mention both FCC and CE, it was part of the subject line. It’d have been hard to make this any clearer.

I told TÜV that we would really appreciate if they wrapped up CE as fast as possible, especially considering the recent delays that they added to our project. They promised me that we’ll have the official CE report by the 18th. They did not promise an ETA regarding the FCC report, but told me that it’ll be ready “soon”.

To be perfectly honest, given the recent complications, we’d like to switch to an alternative certification body, but it’s way too late, and we don’t have any other options in Hungary. Going forward, I’ll be pinging them regularly to try to keep this under control. I believe this will get sorted out soon, and we’ll receive the certification papers shortly.

Our contract manufacturer is eager to start production, and as soon as we receive the reports from TÜV, we’ll start the PCBA of the pilot run units.

Manufacturing progress

Even though the PCBs are not assembled yet, we’re working on assembling and packaging everything else.

There are two small boxes inside of the main UHK box, one containing the USB cable, the other containing the bridge cable, the flip-out feet and their screws, and a lock strip that securely locks together the halves if you choose to use it. We’ve packed 500 of these boxes.

We’ve tested 500 USB cables and 500 bridge cables, and they all worked flawlessly. This is a good sign. Our cable suppliers definitely seem to be on par.

Right now, the case buttons are being pad printed. These are the first samples:

We’ve chosen the top, brighter color sample, even though the difference is hardly noticeable in this picture. Unlike our first pad printing supplier candidate, this supplier seems to perform just as expected. The prints are razor sharp, spotless, precisely positioned, and the correct font is used.

The cases are semi-assembled and we’re waiting for the back stickers to proceed further. We already received 3,000 stickers from the printing factory, but they were shiny instead of matte. Then we received a supposedly corrected batch of 3,000 that were matte, but lacked anti-scratch coating.

Now the printing factory will print yet another 3,000, and we’re hopeful that they’ll get it right this time. This journey is as expensive for them as it is time consuming for us. Once we get the correct back stickers, we’ll apply them to the case, and then glue small rubber feet to the cases,concluding their assembly.

We’re also making progress with the colored cases. This is the first colored case sample of a random test color:

Last but not least, 3,000 UHK boxes have arrived from the printing factory:

Being pre-assembled boxes, they take up quite a lot of floor space. 15 pallets to be exact. Maybe we shouldn't go with preassembled boxes next time.

Firmware progress

I’ve been heavily focusing on the firmware during the last couple of weeks and managed to make quite a lot of progress.

More than anything else, I wanted to make the I2C communication between the keyboard halves rock stable. Placing the bypass capacitors as closely to the IS31FL3731 LED driver ICs as possible made the communication much more robust, but from time to time it halted. The capacitors couldn’t totally negate the parasitic capacitance of the I2C bus, and these LED drivers being as picky as they are, the problem persisted.

Eventually, I managed to find a solution that is described in the AN-686 application note. The core problem is that when I2C communication halts in the midst of the communication, it makes the state machine of the slave that is being addressed wait for further data. Just as suggested by the application note, I clocked through the slave by toggling SCL until SCA went high.

This helped tremendously, and it made communication always recover when disconnecting and reconnecting the keyboard halves, but when using the keyboard over an extended period, the left half eventually became non-responsive.

I could reproduce this issue fairly reliably by making the right keyboard half reenumerate as the bootloader. This interrupted the communication with the left keyboard half, but didn’t unpower it, which always happens when disconnecting it. I also figured that the left KL03 MCU is the culprit because when I rebooted it, the communication always resumed. I needed to fix this issue.

First try: I2C watchdog

I thought that the issue could be solved by implementing an I2C watchdog not only for the right keyboard half but also for the left keyboard half. This proved to be a lot more difficult than anticipated thanks to a bug that I made earlier.

I created a timer interrupt, and put the I2C recovery code into it to reinitialize the I2C driver. Among other things, it called the Init_I2C() function. I realized that when commenting out Init_I2C() within the timer interrupt, the firmware worked, but when uncommenting it, it hit a hard fault. This was seemingly impossible because the hard fault also got hit when I deactivated the timer interrupt, so Init_I2C() wasn’t ever called within it.

I couldn’t figure this out, so I managed to summon Santiago who delved very deep into the issue. It also turned out that the debug version didn’t work at all, so he had to figure this out by looking the disassembled, gcc-optimized ARM assembly code of the firmware.

Santiago finally concluded that the issue was that I defined the I2C_Handler struct within a function so the struct got allocated in the stack, but after returning from the function this variable got deallocated. The problem is, the KSDK still tried to use the struct even after being deallocated, which triggered a hard fault.

The issue didn’t surface when referencing Init_I2C() once across the codebase because the function that contained the definition of the I2C_Handler struct got inlined so it was like being defined in main(), and never got deallocated. But when referring Init_I2C() twice, it wasn’t inlined anymore and the issue surfaced.

After this incident, I’ll surely think twice about making KSDK variables global. For even more details, Santiago created a slide which you’re welcome to study.

Unfortunately, even though my left-side I2C watchdog was running, it still didn’t do the trick. Somehow, communication didn’t recover. Which brings me to my second try.

Second try: hacking the KSDK

At this point, it was down to debugging. I needed to see not only what went over the wire between the MCUs of the left and right halves, but also what was happening inside of the KL03, especially regarding the state of its I2C driver. Using the debug feature of Kinetis Design Studio didn’t work for me as it pauses execution and given the realtime nature of the communication, it doesn’t behave as expected.

Then I tried to make semihosting work which basically allows the microcontroller to dump strings via the debug probe to the console using printf(). Unfortunately, I couldn’t make semihosting work because it crashed the microcontroller for some reason that I couldn’t figure out. Semihosting would also have posed a significant runtime overhead anyways, and it might have made the firmware behave differently, so maybe it’s not a big loss. In any case, I had to find an alternative solution.

Finally, I figured that I’d just use a spare peripheral of the microcontroller to dump relevant data, and I ended up using the SPI peripheral of the KL03. For this, I needed to temporarily repurpose two GPIO pins that were used to scan a row and column of the keyboard matrix. This made the prototype practically useless for touch typing temporarily, but that’s a small price to pay.

At this point, my prototype looked like this:

The halves are interconnected by two spiral cables via an adapter board. The adapter board features a 4P2T switch which allows me to disconnect and reconnect the halves easily for testing purposes. The adapter board also breaks out the wires of the cables. The wires I’m interested in are the SDA and SCL – the clock and data lines of the main I2C bus of the UHK.

Apart from I2C, I also soldered two wires to the MOSI and SCK pins of the SPI peripheral. And finally, I connected the two I2C wires and two SPI wires to my Logic 4 analyzer.

What you can see above is the result of the fixed code. Channels 0 and 1 are the SDA and SCL of the I2C. Channels 2 and 3 are the MOSI and SCK of the SPI. For every byte sent by the master over I2C, the status byte and the received byte get dumped over SPI on the slave. The received byte correctly gets dumped over SPI right after receiving it, which is the way it should be.

The core issue was that the KSDK I2C driver is designed for fixed-size messages, but the UHK messages are variable-sized. They contain a header of a byte length, and a 2 byte CRC16-CCITT checksum. I ended up hacking the KSDK of both the left and right keyboard half to make it deal with our variable-length messages.

The effort that went into making the keyboard halves and the modules communicate reliably is quite incredible. It makes me appreciate already established and widely implemented protocols like TCP/IP that just work. I’m so happy that this critical foundation is in place.

Apart from the above long journey, I implemented loads of fixes. Now we have a pretty changelog, and versioning conventions.

Agent progress

Robi has been really pushing hard recently. I’m impressed by his work, and his efforts came to fruition as Agent is now able to read and write the configuration of the UHK. It’s a wonderful feeling to plug in the UHK, see the configuration appear on the screen, reconfigure it, and merely click a button to save the new configuration to the keyboard.

The smoothness of the user experience resembles the configuration applications of the largest keyboard manufacturers, but all things considered, Agent offers much more sophisticated configuration options than those keyboards.

It’s been a long journey to get there. And wouldn’t it be cool to visualize the development of Agent over its lifetime? As a matter of fact, it’s very much possible with Gource, so let’s take a look at it:

While we’re at it, let’s also take a look at the firmware repo:

There’s still a lot to do, but given the current state of the firmware and Agent, even the pilot run recipients should experience a smooth ride.

Thanks so much for your support!

In our previous update, I elaborated in detail on the manufacturing challenges which have been causing us delays. After publishing our update, it felt awesome that so many of you got in touch with us and expressed your fullest support. We’re glad that you share our pursuit for quality, understand the nature of the delays, and agree that the UHK is worth the wait.

What has been said about the challenges of manufacturing clearly reverberate in this update. After all, who would think that it’s possible to mess up mere stickers (twice), or misunderstand clearly written, basic instructions about the certification process. But it’s the real world, and human errors do happen.

PCB assembly should happen shortly, just after we get the FCC and CE reports from TÜV. Then we’ll quickly assemble the pilot run units and send them out, and the rest of the first batch will follow. We can’t see any major challenges ahead us, we just have to wait longer than anticipated which is something none of us like, but it’s seems that it’s the nature of the hardware business.

Thanks again for your support and understanding! Let’s keep in touch, and we’re excited to talk to you on 2017-11-16.