IRIS Native api for Node.js: all calls synchronous?
Hi,
If I test the Native api for Node.js from the documentation, I noticed (if I'm correct) all methods and calls are synchronous. By default due to the nature of Node.js, there is only one thread of execution and normally all JavaScript methods and all calls should be asynchronous and use either a callback function (the "old way") or promises or the async/await contruct to return their result, e.g.:
- myFunction(params, callbackFunction(response))
- myFunction(params).then(resolveFunction(response))
- async someFunction() { response = await myFunction(params) }
See also this post explaining the concepts.
In the current Native api, I only found methods and functions returning their result synchronously. For a low-load or single user application, this is not a big issue. But for high-load or multi-user applications, as each synchronous Native api call blocks the Node.js execution thread, this can become a severe performance issue. E.g. if you call a class method which performs a longer running embedded SQL query, all users would be blocked until the method call returns.
Am I overlooking asynchronous method variants in the Native api?
Thanks for clarifying!
It has not been updated for a few years, so, yeah, I'm sure it's synchronous.
But I think most of the operations available through that API, should be synchronous. SQL could be asynchronous, but it does not support it.
I think I could make an asynchronous adapter, I have a driver which supports SQL, not async, yet, but I have not have such task yet.
Hi Ward - I too have wondered about the built-in IRIS Native API for Node.js. It does, indeed, appear to be synchronous which is good in that it makes it easy to use and understand, but very bad for the reasons you describe that relate to Node.js concurrency, and effectively unusable as it stands for production multi-user settings.
Conversely, I know from experience that asynchronous versions of the APIs can tie you in horrible knots, and if you need to use locks or transactions, the fact that every concurrent user is sharing the same physical thread of execution makes them pretty much impossible to use. So having async versions of the APIs isn't really the solution either. So does this mean the Node.js API is effectively unusable? No - there's a solution!
First some quick background: I've grappled with and specialised in Node.js integration since the earliest days of Node.js (all the way back to 2010/11!), and long ago concluded that the best solution was a queue/worker architecture, whereby access to IRIS could be limited to Worker processes, within which only a single task would execute in isolation. Remove the concurrency issues of Node.js and you'd have an ideal scenario of being able to use synchronous APIs to access a database such as IRIS without any downsides.
I even wrote an article about this:
https://medium.com/the-node-js-collection/having-your-node-js-cake-and-e...
That article describes my solution in the context of our QEWD.js framework - now, to be fair, QEWD is a somewhat monolithic, all-or-nothing framework/solution which locks you into certain key building-blocks (eg Express as the web framework, mg-dbx for the Node.js APIs to IRIS etc) and particular ways of working.
Recently, however, as a spin-off from other stuff I've been working on, I decided it was time to revisit the core ideas of QEWD and not only modernise its logic to use the latest JavaScript constructs and features where relevant, but also break it down into a set of "building blocks" which would allow people to customise how they use and assemble them - a "buffet menu" instead of a "set meal" as it were.
One such building block is the queue/worker piece, now named QOper8: actually there are two variants - QOper8-cp for Child Process Worker and QOper8-wt for Node.js Worker Thread Workers. Both work identically and have almost identical APIs and configuration. You can read about them here:
https://github.com/robtweed/qoper8-cp
https://github.com/robtweed/qoper8-wt
So these provide a potential queue/worker solution for making the IRIS Native API for Node.js acceptable and workable in a multi-user Node.js environment, but how, you might ask, would you make use of them in practice? What, for example, would be a likely use-case?
Well, the most common scenario for using Node.js in such a multi-user environment is as the back-end of an interactive web application or a REST-based web service. Node.js excels at this: it's exceptionally fast, highly scalable, lightweight in terms of hardware resource demands. A real benefit is you have a single language - JavaScript - describing both the front-end and the back-end logic.
Ask any Node.js practictioner these days what web framework they use and they'll pretty much all now say the same thing - something called Fastify, which, as its name implies, provides an extremely fast web server platform. Furthermore, Fastify has been designed to be easily extensible via a standardised plug-in architecture.
I've therefore used this to create a Fastify plug-in for my QOper8 modules, which, as a result, makes it extremely quick and simple to build out a Fastify/QOper8 back-end whereby incoming HTTP/REST requests can be automatically forwarded to QOper8 which queues and dispatches each request to be handled in isolation within the first available Worker process. You can read about this plug-in, known as qoper8-fastify, here:
https://github.com/robtweed/qoper8-fastify
So then take this one step further: initialise each QOper8 Worker Process by loading the IRIS Native API for Node.js, connecting to your IRIS system and making the APIs available for your handler methods, and you have a high-performance, highly scalable back-end with direct access to your IRIS database, and allowing all your logic to be written in JavaScript (actually you can even use the Native API to invoke ObjectScript functions and procedures on your IRIS system), and safely using the synchronous IRIS Native API for Node.js in a production multi-user environment.
If you want to find out more about how you can implement such a solution, I've written it all up, complete with a simple worked example that you can easily extend to build out your own web/REST back-end solution:
https://github.com/robtweed/qoper8-fastify/blob/master/IRIS.md
Enjoy, and do provide me with feedback!
Hi Rob, this backend architecture is a very good solution and makes the native API really usable for real-world scenarios. Thanks for your contribution which opens up IRIS to the Node.js JavaScript community!
I wonder how many people are actually using Node.js with IRIS? My hunch is, sadly, not many, yet JavaScript is a really great modern alternative language to ObjectScript, and IRIS is potentially a really natural database for JavaScript/Node.js deveopers. How to get IRIS developers to check out and adopt Node.js, and how to get Node.js developers to check out and adopt IRIS? It's an incredibly potent combination for anyone willing to try!
I see it in difficulty explaining of how to start using IRIS with NodeJS (actually with any supported language). It's not even simpler for NodeJS Developers who already familiar with IRIS and the complexity of getting drivers.
When on answer of how to start develop in NodeJS with IRIS, we could answer just install driver with npm and you are ready to go, then probably will be much more NodeJS based projects.
Well of course you can use the mg-dbx interface for IRIS which is available on NPM and which provides equivalent functionality to the Native API interface.
https://github.com/chrisemunt/mg-dbx
Otherwise, in terms of complexity of getting IRIS working with Node.js for multi-user environments, the qoper8-fastify approach I've described makes it about as simple and easy as you can get, and all you need to focus on is the logic for each REST request handler and how it interacts with and uses your IRIS data.
Use glsdb's even higher-level abstraction and it becomes even simpler for a JS/Node.js developer
https://github.com/robtweed/glsdb
So the solutions are all there. It's the means to communicate at appropriate scale that's missing. I've tried relentlessly over recent years, but a single lone voice doesn't get things very far, sadly.
That way requires some sort of familiarity with IRIS, so, be able to load and run the server on the IRIS side. That's not what I expect, from drivers like this. You should be able to connect to any running instance, and it should not require installing something there, at maximum do some security configuration, such as adding users.
Ideally yes, I would agree, but you're up against the publicly available interfaces for IRIS. What would you suggest?
I suggest, that InterSystems, would finally do something, and continue developing and improving their drivers.
I could do such drivers too, and not just NodeJS only, any other languages as well. And I have my own realization for IRIS that way, but it's too simple.
... well they don't need to develop their own. They could actively support and work with third party experts in this area and simply make available suitable public interfaces.
I should point out that if you use the in-process API connection to IRIS from mg-dbx, you don't need to install anything on IRIS - just enable the call-in interface and away you go. You do, of course, need Node.js running on the same physical hardware as IRIS for such a connection, but it's significantly better performance than a networked connection.
If you'd like to add your vote to getting InterSystems to publish on npm the package that adopters need, please consider doing that at https://ideas.intersystems.com/ideas/DPI-I-326
Hi John, I get a broken link in the ideas portal for this idea.