RhinoFit’s Door Access Control (DAC), our proprietary 24/7 access control system, represents our commitment to providing additional features to meet our customers’ needs. However, developing and supporting the DAC has not been without issues. Early adopters that started with the RhinoFit Comport Communicator and Chromebook, who later converted to the DAC, know that new technology comes with growing pains. 2019 has been particularly daunting because of an influx of new demand on RhinoFit’s servers due to on-boarding hundreds of new clients and the high workload the DAC units place on RhinoFit’s infrastructure.
Within the last six months (between April and October), the increased load has exhibited itself in the form of intermittent downtime when attempting to access the RhinoFit application at https://my.rhinofit.ca, failure to process cards for portions of a few days, and intermittent connectivity or functionality of DAC units in the field. While RhinoFit’s up-time still remains over 99%, we understand that any interruption of service is frustrating and, in certain circumstances, unacceptable. Over the last 30-days we have poured over client complaints and requests, as well as server logs, to identify causes and determine potential solutions. Many of the problems were challenging to track down because they occurred in infrastructure outside of our direct control and had never before been an issue – such as public, and Amazon Web Service’s (AWS) internal, Domain Name Servers (DNS) intermittently failing to propagate or deliver correct IP addresses.
It has become obvious that, despite redundancies in our infrastructure already in place, additional redundancies needed to be implemented to prevent further disruptions to service. First, on September 28th, 2019, RhinoFit deployed a dedicated server as an end point for all DAC units to ensure the primary RhinoFit application, and its primary services such as card processing, are functional despite any issues caused by the DAC units; as well as to provide sufficient resources to serve the DAC units’ requests. An email on September 30th, 2019, to all DAC clients addressed an issue with DNS propagation that has since been resolved. As of Saturday, October 19th, 2019, RhinoFit has implemented a series of servers on the back end of a load balancer, explicitly to service the DAC units’ requests, providing additional redundancy and the ability to provide maintenance to said servers without the DAC units experiencing any downtime. Finally, on the week of October 21st, 2019, the RhinoFit network has been re-configured to remove any internal reliance on DNS, in favor of fixed IP addresses, to ensure that any internal DNS issues at AWS no longer propagates into issues for our end users.
We apologize for any inconvenience our clients, or their customers, have experienced during these past months. It is not our intent to frustrate you – we are also a small business and relate wholly with the difficulties you go through every day to keep your businesses profitable and growing. We strive to provide a reliable, feature-rich service, with better customer service than our competitors, at a price point that is accessible for any facility. Unfortunately, we do not always meet our mark. However, we believe that the recent changes discussed above will alleviate many, if not all, reliability issues you may have experienced while using our software and hardware, and thank you for your patience and understanding.