LLM Microservice with Instruction Calling

I describe the idea of implementing interaction with LLM through a concept of microservice with instruction calling. This works great for enterprise application use cases, such as data validation, workflor decisions.

 

TensorFlow.js Microservice

Do you know you could run TensorFlow.js on backend with Node.js? I explain how to build microservice with TensorFlow.js based on sample MobileNet model for image classification. I'm using Skipper Lib JS lib to communicate to RabbitMQ and receive/deliver events. You will learn about the code structure, which can be used as a blueprint for your own JS microservice implementations.

 

Python Numpy Array in ML Services

It can be not obvious how to pass Numpy array across separate services when running ML infra in separate containers. When data preparation service runs in a different container than training service. In this tutorial, I show how to convert Numpy to list and JSON to be able to send it through RabbitMQ message broker and consume it on the receiver side.

 

Running Kubernetes on Oracle Cloud OCI

Oracle Cloud OCI provides a good environment to run your Kubernetes workloads. In this video, I show how to access Kubernetes cluster in OCI, explain artifacts related to the cluster. I show how Skipper API runs on Kubernetes deployed on OCI. Cluster runtime is accessed through cloud shell.

 

MLOps: Sharing Model Across Services

Typically you would want to scale ML model training and ML inference/prediction services separately. This means services should not share the file storage, at least this is true for the Kubernetes cluster environment. I explain how you could transfer files across services using RabbitMQ.

 

RabbitMQ on Kubernetes in Skipper

RabbitMQ works great for event-based microservices implementation. Katana ML Skipper is our open source product, it helps to run workflows and connect multiple services, we are specifically specialized for ML workflows. In this video, I explain how we integrated RabbitMQ and how we run it on Kubernetes cluster. I believe this can be helpful if you are researching how to run RabbitMQ on Kubernetes cluster for your own use cases.

 

Routing Traffic Between FastAPI Pods in Kubernetes

This is a quick tutorial to show how to route traffic between Kubernetes Pods. Both Pods are running FastAPI endpoints. I show how to create Deployment and Service elements for Kubernetes Pod, and how to refer to that service from another Pod to execute HTTP call.

 

FastAPI Running on Kubernetes Pod

Step-by-step tutorial where I explain and show how to run FastAPI app on Kubernetes Pod. I keep it simple. I explain when it makes sense to use multiple containers in a single Pod and when you should put containers into different Pods.

 

Dockerfile and Docker Compose Tutorial for MLOps

This is the tutorial, where I talk about MLOps, explain the difference between Dockerfile and Docker Compose YML definition file. I briefly explain what you should be aware of if planning to move your solution to Kubernetes in the future. I explain in simple words, what is Dockerfile and when Docker Compose is useful. The sample service is based on TensorFlow functionality, where we call model predict function to process serving request.

 

FastAPI Background Tasks for Non-Blocking Endpoints

With FastAPI it is super easy to implement a non-blocking endpoint. This is useful when the endpoint calls logic, which should be executed asynchronously and you don't need to wait for the result, but want to return a response immediately. For example - a service that does logging. We don't want to wait until the log will be written but return the response instantly.

 

FastAPI Endpoint Type with Pydantic

Pydantic library helps to define structured and clear types for FastAPI endpoints. In this video I explain how to define Pydantic type with nested list structure, I show how it works with a live demo. You will get to know how to convert input data into JSON structure. Enjoy!

 

Event-Driven Microservice with RabbitMQ and FastAPI

Event-driven microservices architecture brings scalability and better application structure. In this video, I show a demo based on Web API implementation with FastAPI, Celery. The event is sent to a group of microservices through RabbitMQ broker. Services communicate with each other through RabbitMQ. Model training service calls data service to fetch and prepare data for training.

 

Web API with FastAPI, RabbitMQ and Celery

With Web API you can create access to microservice functionality. In this video, I explain how to create scalable Web API with FastAPI, Celery and RabbitMQ. Celery is responsible to execute async Web API requests and RabbitMQ enables the communication between Web API and microservices.

 

Celery Distributed Task Queue Display with Flower UI

I explain how you can visualize Celery distributed task queue with Flower UI. This is useful when monitoring asynchronous tasks.

 

Celery Distributed Task Queue with FastAPI for Machine Learning

This sample app demonstrates how to implement Celery distributed task queues on top of RabbitMQ broker for Machine Learning model training and data processing. We are using TensorFlow in this example to train the model. API request comes through FastAPI and it is being processed asynchronously by Celery. There is a separate API endpoint to check task status. Multiple requests can be initiated and processed at the same time in parallel. Celery tasks can be monitored using Flower monitoring tool.

 

RabbitMQ RPC with FastAPI

I explain sample app with FastAPI endpoint. RabbitMQ is used as a postman to deliver and return message between endpoint API and backend. Backend code could run on different microservice. Multiple backends can be started for scalability.

 

Rabbit MQ with Docker for Microservices

Rabbit MQ message broker helps to implement event-driven architecture for microservices. Instead of tight coupling multiple services, we can send and subscribe to events. In this video, I explain how to dockerize Rabbit MQ and provide a simple, but complete example of communication through Rabbit MQ.

 

ML Microservice Client with Python and gRPC

In this video, I continue to talk about microservices for ML. I explain how to implement a direct call from the training service to fetch data and run training with model evaluation.

 

ML Microservice with Python and gRPC

I explain how to implement microservice in Python for Machine Learning code. In this example, data processing is done with Pandas, Numpy, and Scikit-Learn libraries. Communication is implemented through gRPC, I explain how to send Numpy array through gRPC.

 

Microservice Approach for Web Development - Micro Frontends

This post is based on my Oracle Code 2018 Warsaw talk. View presentation on slides share:

Wondering what micro frontends term means? Check micro frontends description here. Simply speaking, micro frontend must implement business logic from top to bottom (database, middleware and UI) in isolated environment, it should be reusable and pluggable into main application UI shell. There must be no shared variables between micro frontends. Advantage - distributed teams can work on separate micro frontends, this improves large and modular system development. There is runtime advantage too - if one of the frontends stops working, main application should continue to work.

I have implemented micro frontends architecture with Oracle JET. Source code is available on GitHub repository. There are three applications, two with micro frontends and one is the master UI shell. Both micro frontends are implemented as JET Composite Components. First is hosted on WebLogic, it calls ADF BC REST service in the backend. Second is hosted on Node.JS and returns static data. First micro frontend implements listener, it allows to handle actions from the outside.

When JET application is accessed in your browser, bunch of HTML, JS and CSS files are downloaded from the server. Core idea with micro frontends - instead of loading HTML, JS and CSS for micro frontend from the same host as master app - load it from different host. JET Composite Component rendered inside master application will be downloaded from different host. Not only downloaded, all backend calls should go to that host too, not to the master host. JET Composite Component integration into master application architecture:

This is how it works in practice. Each of these charts is separate JET Composite Component, loaded as micro frontend from different host into master application. We can see that in network monitor. Loader.js scripts for both micro frontends are downloaded from different hosts:

Runtime advantage - if one or multiple micro frontends are down, application continues to run:

JET Composite Component runs on the client, even it is hosted in its own micro frontend. This gives possibility to subscribe to the events happening in the component in the master app and route that event to another micro frontend. In this example, once item is selected in job chart - employees chart (another micro frontend) is filtered:

Technical implementation

Main application must be configured to support remote module loading for JET Composite Component. Read more about it in Duncan Mills blog post - JET Custom Components XII - Revisiting the Loader Script. In short, you should add Xhr config in JET application main.js:

Server where micro frontend is hosted, must set Access-Control-Allow-Origin header.

Main module where both micro frontends are integrated is using JET module component. Each micro frontend in master UI shell is wrapped into JET module. This allows main application to function, even when micro frontend in the module stops:

JET module is initialized from variable, which returns module name:

Jobs module contains Jobs micro frontend - JET Composite Component. It is hosted and WebLogic and calls ADF BC REST in the backend. Component is assigned with listener:

The most important part is in JS script. Here instead of referencing JET Composite Component locally, we load it from remote host. This allows to develop and host micro frontend JET Composite Component on its own:

Listener refers to c2 element and cals the method. Element c2 in the main app relates to second micro frontend:

This component is loaded from another host, from Node.JS:

Important hint - for JET Composite Component to load from remote host, make sure to add .js for JET Composite Component script, as highlighted (refer to source code):