FCC success and development news

Hi there, and welcome to our monthly status update!

TL;DR: We’ve passed FCC certification, and CE is in progress. We’re progressing with assembly, loads of UHK boxes arrived from the printing factory, and the UHK accessory boxes are already packed. The firmware and Agent have matured substantially. We’re waiting for final answers from TÜV to proceed further.

We’re aiming to send out the pilot run UHKs in October, and start the delivery of the rest of the first batch of 2,000 UHKs in November. If you backed us after 2017-07-13 then you’re in the second batch, which is expected to ship in March 2018.

FFC and CE certification

A week ago, we got great news from TÜV:

“All FCC 15 ready and just PASS for Radiated Emission. Producing the report now.”

Then I asked about CE, and to my surprise they told me that they didn’t know that we also needed CE measurements. I searched my emails to see whether I miscommunicated something, but I didn’t. Not only did I mention both FCC and CE, it was part of the subject line. It’d have been hard to make this any clearer.

I told TÜV that we would really appreciate if they wrapped up CE as fast as possible, especially considering the recent delays that they added to our project. They promised me that we’ll have the official CE report by the 18th. They did not promise an ETA regarding the FCC report, but told me that it’ll be ready “soon”.

To be perfectly honest, given the recent complications, we’d like to switch to an alternative certification body, but it’s way too late, and we don’t have any other options in Hungary. Going forward, I’ll be pinging them regularly to try to keep this under control. I believe this will get sorted out soon, and we’ll receive the certification papers shortly.

Our contract manufacturer is eager to start production, and as soon as we receive the reports from TÜV, we’ll start the PCBA of the pilot run units.

Manufacturing progress

Even though the PCBs are not assembled yet, we’re working on assembling and packaging everything else.

There are two small boxes inside of the main UHK box, one containing the USB cable, the other containing the bridge cable, the flip-out feet and their screws, and a lock strip that securely locks together the halves if you choose to use it. We’ve packed 500 of these boxes.

We’ve tested 500 USB cables and 500 bridge cables, and they all worked flawlessly. This is a good sign. Our cable suppliers definitely seem to be on par.

Right now, the case buttons are being pad printed. These are the first samples:

We’ve chosen the top, brighter color sample, even though the difference is hardly noticeable in this picture. Unlike our first pad printing supplier candidate, this supplier seems to perform just as expected. The prints are razor sharp, spotless, precisely positioned, and the correct font is used.

The cases are semi-assembled and we’re waiting for the back stickers to proceed further. We already received 3,000 stickers from the printing factory, but they were shiny instead of matte. Then we received a supposedly corrected batch of 3,000 that were matte, but lacked anti-scratch coating.

Now the printing factory will print yet another 3,000, and we’re hopeful that they’ll get it right this time. This journey is as expensive for them as it is time consuming for us. Once we get the correct back stickers, we’ll apply them to the case, and then glue small rubber feet to the cases,concluding their assembly.

We’re also making progress with the colored cases. This is the first colored case sample of a random test color:

Last but not least, 3,000 UHK boxes have arrived from the printing factory:

Being pre-assembled boxes, they take up quite a lot of floor space. 15 pallets to be exact. Maybe we shouldn't go with preassembled boxes next time.

Firmware progress

I’ve been heavily focusing on the firmware during the last couple of weeks and managed to make quite a lot of progress.

More than anything else, I wanted to make the I2C communication between the keyboard halves rock stable. Placing the bypass capacitors as closely to the IS31FL3731 LED driver ICs as possible made the communication much more robust, but from time to time it halted. The capacitors couldn’t totally negate the parasitic capacitance of the I2C bus, and these LED drivers being as picky as they are, the problem persisted.

Eventually, I managed to find a solution that is described in the AN-686 application note. The core problem is that when I2C communication halts in the midst of the communication, it makes the state machine of the slave that is being addressed wait for further data. Just as suggested by the application note, I clocked through the slave by toggling SCL until SCA went high.

This helped tremendously, and it made communication always recover when disconnecting and reconnecting the keyboard halves, but when using the keyboard over an extended period, the left half eventually became non-responsive.

I could reproduce this issue fairly reliably by making the right keyboard half reenumerate as the bootloader. This interrupted the communication with the left keyboard half, but didn’t unpower it, which always happens when disconnecting it. I also figured that the left KL03 MCU is the culprit because when I rebooted it, the communication always resumed. I needed to fix this issue.

First try: I2C watchdog

I thought that the issue could be solved by implementing an I2C watchdog not only for the right keyboard half but also for the left keyboard half. This proved to be a lot more difficult than anticipated thanks to a bug that I made earlier.

I created a timer interrupt, and put the I2C recovery code into it to reinitialize the I2C driver. Among other things, it called the Init_I2C() function. I realized that when commenting out Init_I2C() within the timer interrupt, the firmware worked, but when uncommenting it, it hit a hard fault. This was seemingly impossible because the hard fault also got hit when I deactivated the timer interrupt, so Init_I2C() wasn’t ever called within it.

I couldn’t figure this out, so I managed to summon Santiago who delved very deep into the issue. It also turned out that the debug version didn’t work at all, so he had to figure this out by looking the disassembled, gcc-optimized ARM assembly code of the firmware.

Santiago finally concluded that the issue was that I defined the I2C_Handler struct within a function so the struct got allocated in the stack, but after returning from the function this variable got deallocated. The problem is, the KSDK still tried to use the struct even after being deallocated, which triggered a hard fault.

The issue didn’t surface when referencing Init_I2C() once across the codebase because the function that contained the definition of the I2C_Handler struct got inlined so it was like being defined in main(), and never got deallocated. But when referring Init_I2C() twice, it wasn’t inlined anymore and the issue surfaced.

After this incident, I’ll surely think twice about making KSDK variables global. For even more details, Santiago created a slide which you’re welcome to study.

Unfortunately, even though my left-side I2C watchdog was running, it still didn’t do the trick. Somehow, communication didn’t recover. Which brings me to my second try.

Second try: hacking the KSDK

At this point, it was down to debugging. I needed to see not only what went over the wire between the MCUs of the left and right halves, but also what was happening inside of the KL03, especially regarding the state of its I2C driver. Using the debug feature of Kinetis Design Studio didn’t work for me as it pauses execution and given the realtime nature of the communication, it doesn’t behave as expected.

Then I tried to make semihosting work which basically allows the microcontroller to dump strings via the debug probe to the console using printf(). Unfortunately, I couldn’t make semihosting work because it crashed the microcontroller for some reason that I couldn’t figure out. Semihosting would also have posed a significant runtime overhead anyways, and it might have made the firmware behave differently, so maybe it’s not a big loss. In any case, I had to find an alternative solution.

Finally, I figured that I’d just use a spare peripheral of the microcontroller to dump relevant data, and I ended up using the SPI peripheral of the KL03. For this, I needed to temporarily repurpose two GPIO pins that were used to scan a row and column of the keyboard matrix. This made the prototype practically useless for touch typing temporarily, but that’s a small price to pay.

At this point, my prototype looked like this:

The halves are interconnected by two spiral cables via an adapter board. The adapter board features a 4P2T switch which allows me to disconnect and reconnect the halves easily for testing purposes. The adapter board also breaks out the wires of the cables. The wires I’m interested in are the SDA and SCL - the clock and data lines of the main I2C bus of the UHK.

Apart from I2C, I also soldered two wires to the MOSI and SCK pins of the SPI peripheral. And finally, I connected the two I2C wires and two SPI wires to my Logic 4 analyzer.

What you can see above is the result of the fixed code. Channels 0 and 1 are the SDA and SCL of the I2C. Channels 2 and 3 are the MOSI and SCK of the SPI. For every byte sent by the master over I2C, the status byte and the received byte get dumped over SPI on the slave. The received byte correctly gets dumped over SPI right after receiving it, which is the way it should be.

The core issue was that the KSDK I2C driver is designed for fixed-size messages, but the UHK messages are variable-sized. They contain a header of a byte length, and a 2 byte CRC16-CCITT checksum. I ended up hacking the KSDK of both the left and right keyboard half to make it deal with our variable-length messages.

The effort that went into making the keyboard halves and the modules communicate reliably is quite incredible. It makes me appreciate already established and widely implemented protocols like TCP/IP that just work. I’m so happy that this critical foundation is in place.

Apart from the above long journey, I implemented loads of fixes. Now we have a pretty changelog, and versioning conventions.

Agent progress

Robi has been really pushing hard recently. I’m impressed by his work, and his efforts came to fruition as Agent is now able to read and write the configuration of the UHK. It’s a wonderful feeling to plug in the UHK, see the configuration appear on the screen, reconfigure it, and merely click a button to save the new configuration to the keyboard.

The smoothness of the user experience resembles the configuration applications of the largest keyboard manufacturers, but all things considered, Agent offers much more sophisticated configuration options than those keyboards.

It’s been a long journey to get there. And wouldn’t it be cool to visualize the development of Agent over its lifetime? As a matter of fact, it’s very much possible with Gource, so let’s take a look at it:

While we’re at it, let’s also take a look at the firmware repo:

There’s still a lot to do, but given the current state of the firmware and Agent, even the pilot run recipients should experience a smooth ride.

Thanks so much for your support!

In our previous update, I elaborated in detail on the manufacturing challenges which have been causing us delays. After publishing our update, it felt awesome that so many of you got in touch with us and expressed your fullest support. We’re glad that you share our pursuit for quality, understand the nature of the delays, and agree that the UHK is worth the wait.

What has been said about the challenges of manufacturing clearly reverberate in this update. After all, who would think that it’s possible to mess up mere stickers (twice), or misunderstand clearly written, basic instructions about the certification process. But it’s the real world, and human errors do happen.

PCB assembly should happen shortly, just after we get the FCC and CE reports from TÜV. Then we’ll quickly assemble the pilot run units and send them out, and the rest of the first batch will follow. We can’t see any major challenges ahead us, we just have to wait longer than anticipated which is something none of us like, but it’s seems that it’s the nature of the hardware business.

Thanks again for your support and understanding! Let’s keep in touch, and we’re excited to talk to you on 2017-11-16.

Just before the pilot run

Hi there, and welcome to our monthly status update!

TL;DR: We’re making rapid progress with the assembly process. The firmware and Agent have matured significantly and become truly cross-platform. We have to make a minor design change on the PCB that will set us back by 2 weeks. Batch 1 has sold out and batch 2 will ship starting December.

You can always check out the expected delivery date and update your shipping address on your Crowd Supply account page.

Batch 1 has sold out

Given the uninterrupted pre-sales period since the end of our campaign, the first batch of 2,000 UHKs has just sold out. That means that going forward, new orders will fall into batch 2 which can be expected to ship in December.

Your orders will still be served on a first-come, first-served basis, but this delay between batch 1 and batch 2 is inevitable as we ramp up production.

Starting from batch 3 we expect to have a decent stock, so we’ll be able to provide you UHKs without any delay.

The inner assemblies are coming together

The inner assembly is comprised of a number of parts that reside inside the UHK case.

16,000 metal guides are already made. We'll have to press-fit the pins into half of them. Speaking of the pins, we’ve further optimized them. We’ve switched to using pins with spherical ends and a hardened surface, resulting in a heavenly smooth, yet solid merge-split experience. Perfecting these guides was an art unto itself.

600 pairs of plates have been manufactured and zinc-plated.

We’ve learned that seemingly mundane parts like keycap stabilizers can take surprisingly long time to assemble. There are two identical left and two right parts that slip together, then a metal wire is inserted into them. Thankfully, we’ve already finished assembling 5,500 of these.

The above results in assembled plates like these.

So far, we’ve assembled 150 pairs of plates, and 50 pairs of those will be used for the pilot run. These plates will travel to our PCBA contract manufacturer who will solder them together with the PCBs. Afterwards, the guides will be screwed to the plates, completing the inner assemblies.

Outer assemblies and miscellaneous parts

As you can imagine, one of the most complicated parts are the plastic cases. So far, we’ve manufactured 150 pairs of bottom cases, and 2,000 pairs of case buttons.

The molds for the upper cases are being tweaked and we expect them to be perfected within 1-2 weeks, at which point we’ll put them to good use and manufacture 150 pairs of top cases.

Apart from the above, there are still a fair number of parts that go with the outer assemblies including the magnets, magnet-counter parts, rubber feet, inserts, back stickers, and a couple of screws.

Since our last update, we’ve also ordered 2,000 USB cables and bridge cables. Fun fact: the total length of these USB cables is 3.6 kilometers, or 2.23 miles.

Firmware progress

The firmware already works pretty well, but the configuration parser was very basic a month ago, only capable of parsing layers. It was clearly incomplete because it should parse not only a single layer, and not even a single keymap, but the fully-fledged binary configuration containing a set of layers, macros, and various items.

To my surprise, I received an email about a month ago from Eric Tang. Given his passion for keycaps, we’ve already exchanged numerous emails with Eric, but this email was different.

Eric asked me whether we have an internship program. He caught me by surprise and even though we’ve never had an internship program, I asked him what was on his mind exactly. As it turned out, Eric has written his own keyboard firmware based on AVRs and LUFA, making him the perfect candidate to develop the configuration parser of the UHK.

Since then, Eric has made tons of progress and now the firmware is able to parse almost the whole UHK configuration except macros. He’s now working on making the LED display show the abbreviation of the actual keymap, and then he’ll finish off macros, and develop the macro engine.

I feel very fortunate to have been contacted by Eric, and am having a great time working with him. The configuration parser shall be implemented soon!

Agent progress

Quite a few things have happened with Agent.

Right after Eric started working on the configuration parser, he couldn’t make Agent work on his Mac. I originally believed that it’s a privilege escalation issue, but Robi investigated further and realized that it’s a much deeper problem.

As it turned out, libusb, the library we used to make Agent talk to the UHK via USB isn’t supported on the Mac anymore. Apparently, recent OSX versions implement a new version of the USB subsystem which broke libusb.

Luckily, Robi found hidapi, the perfect alternative for libusb. In his true fashion, he didn’t waste his time, and ported Agent to use from libusb to hidapi. As a result Agent now runs perfectly on all the three major OSes.

Now, there’s only portability problem to be solved. blhost, the command line utility that updates the firmware of the UHK also utilizes libusb and it segfaults on OSX now that libusb is broken on Mac. Unfortunately, NXP doesn’t seem to care but fortunately, Santiago made me aware of pyKBoot, a python command line application that speaks the kboot protocol just like blhost, but which uses hidapi instead of libusb. I’ve toyed with pyKBoot a bit, and looks like it’s a viable alternative to blhost.

I think we can use pyKBoot but I’m not fond of it because we’ll have to package it as a standalone, self-contained executable, possibly using PyInstaller which will likely make the bundle size of Agent considerably larger than optimal. I’d really like to replace pyKBoot eventually with something more native to Agent. I think the best way forward is to port pyKBoot from Python to TypeScript later down the line.

Speaking of other parts of Agent, Robi has also implemented an auto-update mechanism. We have yet to refine it, but it seems to be already working.

Józsi also implemented a more general rendering mechanism for Agent, enabling it to render both ANSI and ISO keymaps according to your UHK version.

Electronics issues

Few days ago, I started to heavily test my backlit UHK prototype and found a serious issue. For some reason, the communication between the keyboard halves interrupted from time to time and eventually died.

Frustrated but determined, I delved into the issue and figured that it was caused by the LED drivers. When I disabled the LEDs, the prototype was rock stable. I further narrowed the cause, and found that the issue only surfaced when I enabled one of two LEDs, the LED of the right Alt key or the LED of the comma key.

I couldn’t make sense of this, so I got in touch with Integrated Silicon Solution Inc., the manufacturer of the LED driver ICs that we use. As it turned out, we didn’t place the bypass capacitor of the LED driver close enough to the IC. I soldered a through-hole capacitor as close as I could to the LED driver on my prototype and it solved the issue in no time.

Even though we don’t provide a backlit UHK version yet, many of you have expressed your desire to solder LEDs to your UHKs, so this is a big deal. Even more importantly, we already utilize the left side LED driver to drive the LED display and even though it didn’t produce such issues in itself, I fear that such issues could surface on a quite a few UHKs out of 2,000 UHKs, so this issue must be resolved.

Admittedly, this sucks, but thankfully, it’s a very minor PCB redesign which I’ll implement in about a day. We will, however, have to get new prototype PCBs made, assembled, and tested, so it will translate to a 2 week delay.

If there’s someone to blame, I surely am. I should have tested this way earlier. I’m sorry about this, but I’m also thankful because TUV hasn’t started testing our prototypes for the certification process and our PCBA contractor hasn’t started to manufacture the PCBs for the pilot run yet. I sent them notice just at the last minute. If I didn’t, we would have lost a lot of money due to this issue.

All things considered, even despite the PCB redesign, we’re getting very close. Pilot run notification emails will be sent out soon, followed by 50 UHKs, and followed by the rest of the 1,950 UHKs of batch 1. We cannot wait to finally ship!

Thank you for reading this update! As always, we’ll be keeping you updated on a monthly basis via our blog and newsletter.

Talk to you on 2017-08-17!