Motivation
In recent days, Google’s liberally licensed UI toolkit Flutter is getting a lot of attention. While the original 1.0 release of Flutter only supported Android and iOS devices, the release 2.0 from March 2021 added official support for Web and Desktop as well. Additionally, there is an increased interest in running Flutter on embedded devices to implement embedded HMIs. Also Google recognized this and added “embedded devices” as an official target for Flutter lately, even relatively prominent on Flutter’s homepage. There are various very interesting projects underway. To name one have a look at this Sony project.
Porting from Qt Quick / QML to Flutter?
Here at basysKom, one of our primary fields of interest are HMIs on embedded devices. That is why we are following Flutter’s development in this area very closely.
Recent experiments with internal projects showed that those developments are very promising. In this post my colleague Jeremias explained already last year that Flutter applications can run on i.MX6 devices.
This year we ported our industrial HMI showcase from Qt Quick to Flutter to see how this works out. The showcase is not very complex. Nevertheless, the project underlined that developing embedded HMIs with Flutter is quite practical and that the ecosystem and tooling is already very pleasant and mature. We were able to run this showcase application on various platforms like Linux, Android, Web and an i.MX8 device without too much hassle.
The missing part: Plugins
However one thing was still missing. The original application is a Qt Quick-based HMI using Qt OPC UA to communicate with an OPC UA server. There is no direct OPC UA support in Flutter/Dart. To deal with this we originally used a Node.js gateway which bridged OPC UA to HTTP and Websockets (which works like a charm with Flutter), so we were able to connect the Flutter application to the backend using these technologies. But of course it would be nicer to not take this detour and directly communicate via OPC UA.
As there is no Dart implementation of OPC UA the question was whether it is possible to call native code from Flutter and use an existing library like open62541 to implement the connection to our backend. Not only would this be nice for our showcase, but of course binding to native code is a rather important requirement for many applications, especially in the field of embedded HMIs. The two most obvious use-cases are the reuse of existing libraries and the implementation of performance-critical parts using adequate (meaning fastest possible) languages.
Fortunately, the release of Dart 2.12 / Flutter 2.0 brought us, in addition to the support of more platforms, also a library called dart:ffi that makes foreign function interfaces possible for Dart and Flutter. This is an official part of Dart now, and it is provided and maintained by the Dart team.
After doing some experiments with dart:ffi and porting some parts of our showcase, we decided to share our experiences and write this blog post. Interested readers can play with a small example that I created; you can find in our GitHub.
It is based on an existing example server shipped with the Qt OPC UA module to have a simple server providing some values for our UI.
In the following sections I will briefly present my findings and walk through the essential aspects of the implementation.
Concept
To get started with dart:ffi you can have a look at the official documentation and good examples. Getting an idea of how it works and trying out the examples yourself does not take too long and is quite interesting already.
Having a closer look at the examples, you will notice that they all work in a synchronous manner: the native code is called from Dart synchronously and there are no things running in the background in parallel. Transferred to our task, in which we want to receive data updates from the backend, this means that we would need to implement some polling in order to get new data from the backend. In fact, this is the first thing that I tried. Implementing this pattern is rather straightforward and can be adapted from the examples. However, it is important to note at this point that the Dart profiler can have an influence on the execution of syscalls in debug mode. This has caused a strange problem for me when querying values. For more details on this, have a look at my question and the corresponding answer in this issue.
One important characteristic of OPC UA is the client server communication consisting of service calls with asynchronous replies. For HMI applications, the most important asynchronous operation is monitoring node attributes in the server for changes by using server-side polling and only reporting changed values to the client. For us, one important question was whether it is possible with dart:ffi to implement this asynchronously as well.
Searching for this topic very quickly leads to this interesting discussion on GitHub. The bottom line is that it is possible to implement asynchronous callbacks from C to Dart. Specifically this means that you can start a background thread in your native library that deals with OPC UA and calls back into Dart whenever there are updates. Because of Dart’s design this is not a matter of course. For concurrent tasks Dart uses so called isolates. A very nice summary of Dart isolates can be found here but the main issue is that there must only be one thread per isolate and having a background thread from your native library manipulating state in Dart’s main isolate would break a fundamental assumption of Dart, so it is not an option here.
However, the ffi team came up with a solution using Dart’s ports mechanism that is originally used to implement a message passing mechanism between isolates. In this use case the mechanism is used to put a so-called microtask into the queue of Dart’s event loop. So the callback does not run in the background thread, but a message to run some specific code is sent to Dart. When the microtask is then executed, it runs in the thread of Dart’s main isolate. In practice the implementation is a bit tricky and not as straightforward as the synchronous way of interoperation. However, the ffi team is planning to improve this in the future to make the syntax more concise and reduce the amount of required boilerplate code.
To illustrate the idea behind this process, here is a simplified sequence diagram highlighting the most important points and how the various steps relate to each other.
Setup
If you did not use Flutter before you can install it with only a few steps (see the official documentation). The installation already includes dart:ffi (and the package ffi which contains useful utility functions) so no further steps are necessary.
If you want to use dart:ffi in your Flutter application, you have to create a Flutter plugin. As mentioned before, ffi is a Dart library which can also be used without flutter. In this case you would create a regular Dart package. However, to create a Flutter plugin you can use the flutter command line tool and give the target platforms as a parameter like this:
flutter create --platforms=linux,android --template=plugin waterpump_ffi_plugin Conveniently, this command already generates a sample Flutter application that uses the plugin in the example subdirectory. Note that the dependency in the respective pubspec.yaml file points to the parent directory instead of a pub.dev package.
The next step in order to make use of a native library is to integrate the library into your plugin. To do this you have to adapt the respective project files, which is a platform dependent task that you need to carry out for all your supported platforms. I tested my application on the android and linux platforms. For android you can follow the steps from the documentation. For linux there is no precise documentation yet but if you are familiar with CMake you can add the necessary stuff in linux/CMakeLists.txt. I did my tests using a shared library, but according to the documentation you can also use static ones.
You have to decide where to put your native code. I created a directory native_code in my project directory. But be aware that if you are planning to also compile for iOS devices (which I didn’t test) you should put your code into the ios directory because the toolchain requires this and the other toolchains are flexible enough in this concern.
To make use of asynchronous callbacks, you also need to get the dart sdk and add some header and source files from sdk/runtime to your build. Another important point is that you have to use a C++ compiler if you adhere to the asynchronous examples provided by the Dart
team because they are using lambdas to hand over the work to be done from the library to Dart (see Function MyCallbackNonBlocking here).
This in turns requires you to deal with the name mangling introduced by the C++ compiler, otherwise dart:ffi will not find your functions. To do this you can mark your functions with a macro like (see https://flutter.dev/docs/development/platform-integration/c-interop):
#define EXTERN_C extern "C" __attribute__((visibility("default"))) __attribute__((used))
EXTERN_C void myFunction(); To integrate open62541, you need to also get those sources. I tested my application using the latest stable version 1.2. As part of my first steps I always compiled everything manually and just passed the path to the .so file in my dart code. Of course this is not sufficient for a platform independent solution using different toolchains. So what you have to do here is to add open62541 as a dependency to your CMake files. As mentioned before, I tested my application only with shared libraries so in the case of open62541 you have to change that first in open62541 CMake file by setting the respective option to:
option(BUILD_SHARED_LIBS "Enable building of shared libraries (dll/so)" ON) I also had issues with the address sanitizer that I did not look into yet when using flutter run (build was working though) so I had to disable this by commenting out some lines:
# set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} ${SANITIZER_FLAGS}")
# set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${SANITIZER_FLAGS}") For more details about the setup have a look at the example application.
Implementation
In this section I would like to go briefly through some details of my example implementation. Since we are about to use a foreign function interface to establish interoperation between C and Dart there are two perspectives here, the C and the Dart side. We will start with the native C/C++ library.
The native library
As a starting point I used the open62541 example for asynchronous subscriptions that can be found here and adapted that to my needs. In our example you can find the code in native_code/waterpump_ffi_lib.{h,cpp}.
The first thing we need to adapt is the initialization. As this demo is only an experiment and by no means a generic open62541 wrapper, our C library has to contain some specifics regarding the business logic. In concrete terms this means that we need to add the subscriptions for the nodes we are interested in. I limited this to a very minimum that we want to use in our Flutter application: The current time, the fill state of the tanks and the target fill of tank 2.
Whenever there is an update to one of those nodes, a handler function (for example handler_currentTimeChanged) is called, which in turn triggers the callback to dart. These subscriptions and handler functions need to be initialised, which is done in the function stateCallback in this example.
The second thing is to move the open62541 endless loop from the main function to a regular function and provide a public function that can be called from the Dart side to start the background operation. For this I used a std::thread and provided the function that contains the open62541 endless loop.
The final steps are to define the library interface for Dart (see functions marked with EXTERN_C) and to add things necessary for asynchronous callbacks from the ffi examples. I adapted those to make the code a bit more generic. For a real project one would of course put much more thought into this part; but to show how it works in general, this should be fine.
What you can see in the handler functions is that everything that is passed over to Dart is allocated on the heap. So what must be taken care of in the context of the handler functions is memory management. Memory that is allocated here must be cleared later after everything has been processed, which means, that we need to do this in Dart. More on this point in the next section about the Dart side.
In the concept section we briefly looked at how the mechanism for asynchronous callbacks works in general. Here I would like to add some technical details. First of all the ffi team introduced the typedef
typedef std::function<void()> Work; which is an abstraction used to hold a lambda that scopes a pointer to the data and a pointer to the Dart function to be called from the main Dart thread – the actual callback that will process the updated data on the Dart side. The body of the lambda just calls the callback function with the data as parameter. This lambda is then passed over to notifyDart which puts the lambda in a Dart_CObject and uses the function Dart_Post_CObject_DL (both of which are from the Dart SDK mentioned in the setup section) to actually add it as a microtask to Dart’s event queue.
This whole construct might seem a bit confusing at first. But, as mentioned before, hopefully the ffi team will improve this in the future so it will get a little more convenient to use.
The Dart side
On the Dart side there are also some initialization steps necessary that are implemented in the class that was added by the flutter create command.
These are:
- Initialization of the Dart API, which is implemented as a synchronous call to a C function, which in turn calls
Dart_InitializeApiDLfrom the Dart SDK - Initialization of the Dart port that is used to communicate between the background and the Dart thread
- Looking up the functions from the C library used by Dart and storing a corresponding Dart function
- Registering the callback functions, meaning that the respective Dart callback functions are communicated to the C library
- Starting the background thread using the function mentioned in the previous section
ffigen
This is a good place to briefly discuss another tool from the ffi ecosystem called ffigen. ffigen is a tool to generate bindings for Dart. It uses (besides other parameters) C header files to create a Dart class containing all the boilerplate necessary.
This is a very nice tool, but unfortunately it does not cover every use case yet. In the context of my first (exclusively synchronous) experiments with open62541 I tried to automatically generate appropriate bindings using ffigen, but the tool does not handle inline functions – of which open62541 makes heavy use.
So thinking about that and asking the very supportive team behind ffigen there are two options to deal with this. Either write a tool (or extend ffigen itself, as proposed here) to handle this, or manually create a thin wrapper in C that implements necessary abstractions and at least use ffigen to create bindings for that.
For the asynchronous example covered in this post, we can also not benefit from the tool because we are not using plain C here. In the future there might be support for C++ as well as discussed here. So ffigen is definitely an interesting project you should keep on your watchlist but we will not discuss it any further in the context of this post.
Using the plugin
Apart from the initialization, two things need to be done in our binding class.
The first is implementing the callbacks to process the data updates and providing the data to the actual app using your plugin. As mentioned before, one important thing to be aware of when implementing the callbacks is memory management. Data has been allocated in the C handler functions and is eventually used in the Dart callbacks we are talking about here.
So after the data has been processed, we need to free the respective memory. This can be done with dart:ffi using the top-level constant malloc which is an Allocator providing the method free to release memory on the heap. In the callbacks we take the data from the heap, create appropriate Dart data structures or variables and release the allocated memory.
After that the plugin needs a way to distribute the data to your app. For this I used Dart streams which can be used conveniently by stateful widgets in your Flutter app. But of course there are plenty of more sophisticated options in the Flutter universe to implement this. From here you can just instantiate the plugin in your app and listen to the respective streams. But don’t forget to do that within setState()
Conclusion
In summary we can say that in addition to the generally pleasant experience using Flutter and its ecosystem of tools and packages we were positively surprised with the maturity of dart:ffi. Of course this kind of task is much more straightforward with a framework like Qt – because there is no language barrier – but it is nice to see that it is already possible to combine Flutter with native code and profit from Flutter’s tooling, its platform support and its liberal BSD license.
Although this demo project is only an example, and real projects will probably bring up some more challenges, it is interesting to see that it works in general and that more work is put into both: better support for more platforms and the interoperation to native code.
