Tools Resources

DZone's Featured Tools Resources

Using Unblocked to Fix a Service That Nobody Owns

By John Vester

CORE

Working in technology for over three decades, I have found myself in a position of getting up to speed on something new at least 100 times. For about half of my career, I have worked in a consulting role, which is where I faced the challenge of understanding a new project the most. Over time, I established a personal goal to be productive on a new project in half the time it took for the average team member. I often called this the time to first commit or TTFC. The problem with my approach to setting a TTFC record was the unexpected levels of stress that I endured in those time periods. Family members and friends always knew when I was in the early stages of a brand-new project. At the time, however, since I always wanted to provide my clients with the best value for the rate they agreed to pay for my services, there really wasn’t any other option. Recently, I discovered Unblocked … which provides the possibility to crush TTFCs I had set on past projects. About Unblocked Unblocked, which is still in beta, at the time of this writing, is focused on removing the mysteries in your code. The AI platform trains on all the information about your project and then allows you to ask questions and get answers about your project and codebase. It absorbs threads stored within instant messaging, pull requests, source code, and bugs/stories/tasks within project manager software. Even project information stored in content collaboration solutions can be consumed by Unblocked. Information from these various sources is then cataloged into a secured repository owned and maintained by Unblocked. From there, a simple user interface allows you to ask questions…and get answers fast… in a human-readable format. Use Case: The Service Owned By No One The idea of taking ownership of a service or solution that is owned by no one has become quite common as API adoption has skyrocketed. Services can be initialized to meet a shared need by contributors from various groups within the organization. This can be an effective approach to solve short-term problems, however, when there is no true service owner, the following long-term challenges can occur: Vulnerability mitigation – who is going to address vulnerabilities as they surface? Bug fixes and enhancements – who is going to fix or further extend the service? Tooling updates – who will handle larger scale migrations, like a change in CI/CD tooling? Supportability – who is responsible for answering general questions posed by service consumers? I ran into these exact issues recently, because my team inherited a service that was effectively owned by no one. In fact, there were features within the service that had very little documentation, except the source code itself. The challenge for our team was that a bug existed within the original source code and we weren’t sure what the service was supposed to be doing. Efforts to scan completed tickets in Jira or even Confluence pages would result in incomplete and incorrect information. I attempted to perform searches against the Slack instant messaging service, but it appeared that the chat history around these concepts had long since been removed as a part of corporate retention policies. Getting Started with Unblocked The Unblocked platform can be used to reduce an engineer’s TTFC by simply selecting the source code management system they wish to use: After selecting which source code repositories you wish to use, you have the opportunity to add integrations with Slack and Jira as shown below: Additional integrations can be configured from the Unblocked dashboard: Confluence Linear Notion Stack Overflow After setup, Unblocked begins the data ingestion and process phase. The amount of time required to complete this step is largely dependent on the amount of data that needs to be analyzed. At this point, one of the following client platforms can be prepared for use: Unblocked Client for macOS Unblocked IDE Plug-in for Visual Studio Code Unblocked IDE Plug-in for any JetBrains IDE (IntelliJ, PyCharm, and so on) There is also a web dashboard that can be accessed via a standard web browser. Where Unblocked Provides Value I decided to use the web dashboard. After completing the data ingestion and processing phase, I decided to see what would happen if I asked Unblocked “How does the front end communicate with the back end?” Below is how the interaction appeared: When I clicked the block-patterns.php file, I was taken directly to the file within the connected GitHub repository. Diving a little deeper, I wanted to understand what endpoints are available in the backend. This time I was provided the result from an answer that had been asked 11 days earlier. What is really nice is that the /docs URI was also provided, saving me more time in getting up to speed. I also wanted to understand what changes had been made to the backend recently. I was impressed by the response Unblocked provided: For this answer, there were five total references included in the response. Let’s take a look at several of these references. Clicking the first reference provided information from Github: The second reference provided the ability to download Markdown files from the Git source code management: The experience was quite impressive. By asking a few simple questions, I was able to make huge progress in understanding a service that is completely new to me in a matter of minutes. Conclusion The “service owned by no one” scenario is more common now than at any other point in my 30+ year career in technology. The stress of having issues to understand and fix – without any documentation or service owner expertise – does not promote a healthy and productive work environment. My readers may recall that I have been focused on the following mission statement, which I feel can apply to any IT professional: “Focus your time on delivering features/functionality that extends the value of your intellectual property. Leverage frameworks, products, and services for everything else.” - J. Vester Unblocked supports my personal mission statement by giving software engineers an opportunity to be productive quickly. The platform relies on a simple interface and AI-based process to do the hard work for you, allowing you to remain focused on meeting your current objectives. By asking a few simple questions, I was able to gain valuable information about the solutions connected to Unblocked. In a world where it can be difficult to find a subject matter expert, this is a game changer – especially from a TTFC perspective. While updating my IntelliJ IDEA client I realized there is even an Unblocked plug-in that I could have utilized just as easily too! The same good news applies to users of Visual Studio Code. This functionality allows engineers to pose questions to Unblocked without leaving their IDE. The best part is that Unblocked is currently in an open beta, which means it is 100% free to use. You can get started by clicking here. Take Unblocked for a test drive and see how it holds up for your use case. I am super interested in hearing about your results in the comments section. Have a really great day! More

Demystifying Static Mocking With Mockito

By Arnošt Havelka

CORE

These days, writing tests is a standard part of development. Unfortunately, we need to deal from time to time with a situation when a static method is invoked by a tested component. Our goal is to mitigate this part and avoid third-party component behavior. This article sheds light on the mocking of static methods by using the "inline mock maker" introduced by Mockito in the 3.4 version. In other words, this article explains Mockito.mockStatic method in order to help us with unwanted invocation of the static method. In This Article, You Will Learn How to mock and verify static methods with mockStatic feature How to setup mockStatic in different Mockito versions Introduction Many times, we have to deal with a situation when our code invokes a static method. It can be our own code (e.g., some utility class or class from a third-party library). The main concern in unit testing is to focus on the tested component and ignore the behavior of any other component (including static methods). An example is when a tested method in component A is calling an unrelated static method from component B. Even so, it's not recommended to use static methods; we see them a lot (e.g., utility classes). The reasoning for avoiding the usage of static methods is summarized very well in Mocking Static Methods With Mockito. Generally speaking, some might say that when writing clean object-orientated code, we shouldn’t need to mock static classes. This could typically hint at a design issue or code smell in our application. Why? First, a class depending on a static method has tight coupling, and second, it nearly always leads to code that is difficult to test. Ideally, a class should not be responsible for obtaining its dependencies, and if possible, they should be externally injected. So, it’s always worth investigating if we can refactor our code to make it more testable. Of course, this is not always possible, and sometimes we need to mock static methods. A Simple Utility Class Let's define a simple SequenceGenerator utility class used in this article as a target for our tests. This class has two "dumb" static methods (there's nothing fancy about them). The first nextId method (lines 10-12) generates a new ID with each invocation, and the second nextMultipleIds method (lines 14-20) generates multiple IDs as requested by the passed argument. Java @UtilityClass public class SequenceGenerator { private static AtomicInteger counter; static { counter = new AtomicInteger(1); } public static int nextId() { return counter.getAndIncrement(); } public static List<Integer> nextMultipleIds(int count) { var newValues = new ArrayList<Integer>(count); for (int i = 0; i < count; i++) { newValues.add(counter.getAndIncrement()); } return newValues; } } MockedStatic Object In order to be able to mock static methods, we need to wrap the impacted class by "inline mock maker." The mocking of static methods from our SequenceGenerator class introduced above is achievable by MockedStatic instance retrieved via Mockito.mockStatic method. This can be done as: Java try (MockedStatic<SequenceGenerator> seqGeneratorMock = mockStatic(SequenceGenerator.class)) { ... } Or Java MockedStatic<SequenceGenerator> seqGeneratorMock = mockStatic(SequenceGenerator.class)); ... seqGeneratorMock.close(); The created mockStatic instance has to be always closed. Otherwise, we risk ugly side effects in next tests running in the same thread when the same static method is involved (i.e., SequenceGenerator in our case). Therefore, the first option seems better, and it is used in most articles on this topic. The explanation can be found on the JavaDoc site (chapter 48) as: When using the inline mock maker, it is possible to mock static method invocations within the current thread and a user-defined scope. This way, Mockito assures that concurrently and sequentially running tests do not interfere. To make sure a static mock remains temporary, it is recommended to define the scope within a try-with-resources construct. To learn more about this topic, check out these useful links: The official site, JavaDoc site and GitHub repository. Mock Method Invocation Static methods (e.g., our nextId or nextMultipleIds methods defined above) can be mocked with MockedStatic.when. This method accepts a functional interface defined by MockedStatic.Verification. There are two cases we can deal with. Mocked Method With No Argument The simplest case is mocking a static method with no argument (nextId method in our case). In this case, it's sufficient to pass to seqGeneratorMock.when method only a method reference (see line 5). The returned value is specified in a standard way (e.g., with thenReturn method). Java @Test void whenWithoutArgument() { try (MockedStatic<SequenceGenerator> seqGeneratorMock = mockStatic(SequenceGenerator.class)) { int newValue = 5; seqGeneratorMock.when(SequenceGenerator::nextId).thenReturn(newValue); assertThat(SequenceGenerator.nextId()).isEqualTo(newValue); } } Mocked Method With One or More Arguments Usually, we have a static method with some arguments (nextMultipleIds in our case). Then, we need to use a lambda expression instead of the method reference (see line 5). Again, we can use the standard methods (e.g. then, thenRetun, thenThrow etc.) to handle the response with the desired behavior. Java @Test void whenWithArgument() { try (MockedStatic<SequenceGenerator> seqGeneratorMock = mockStatic(SequenceGenerator.class)) { int newValuesCount = 5; seqGeneratorMock.when(() -> SequenceGenerator.nextMultipleIds(newValuesCount)) .thenReturn(List.of(1, 2, 3, 4, 5)); assertThat(SequenceGenerator.nextMultipleIds(newValuesCount)).hasSize(newValuesCount); } } Verify Method Invocation Similarly, we can also verify calls of the mocked component by calling seqGeneratorMock.verify method for the method reference (see line 7) Java @Test void verifyUsageWithoutArgument() { try (MockedStatic<SequenceGenerator> seqGeneratorMock = mockStatic(SequenceGenerator.class)) { var person = new Person("Pamela"); seqGeneratorMock.verify(SequenceGenerator::nextId); assertThat(person.getId()).isEqualTo(0); } } Or the lambda expression (see line 6). Java @Test void verifyUsageWithArgument() { try (MockedStatic<SequenceGenerator> seqGeneratorMock = mockStatic(SequenceGenerator.class)) { List<Integer> nextIds = SequenceGenerator.nextMultipleIds(3); seqGeneratorMock.verify(() -> SequenceGenerator.nextMultipleIds(ArgumentMatchers.anyInt())); assertThat(nextIds).isEmpty(); } } Note: please be aware that seqGeneratorMock doesn't provide any value here, as the static methods are still mocked with the defaults. There's no spy version so far. Therefore, any expected return value has to be mocked, or the default value is returned. Setup The mockStatic feature is enabled in Mockito 5.x by default. Therefore, no special setup is needed. But we need to set up Mockito for the older versions (e.g., 4.x). Mockito 5.x+ As it was already mentioned, we don't need to set up anything in version 5.x. See the statement in the GitHub repository: Mockito 5 switches the default mockmaker to mockito-inline, and now requires Java 11. Old Mockito Versions When an older version is used, and we use the mock-inline feature via mockStatic then we can see an error like this: Plain Text org.mockito.exceptions.base.MockitoException: The used MockMaker SubclassByteBuddyMockMaker does not support the creation of static mocks Mockito's inline mock maker supports static mocks based on the Instrumentation API. You can simply enable this mock mode, by placing the 'mockito-inline' artifact where you are currently using 'mockito-core'. Note that Mockito's inline mock maker is not supported on Android. at com.github.aha.poc.junit.person.StaticUsageTests.mockStaticNoArgValue(StaticUsageTests.java:15) at java.base/java.lang.reflect.Method.invoke(Method.java:580) at java.base/java.util.ArrayList.forEach(ArrayList.java:1596) at java.base/java.util.ArrayList.forEach(ArrayList.java:1596) Generally, there are two options to enable it for such Mockito versions (see all Mockito versions here). Use MockMaker Resource The first option is based on adding <project>\src\test\resources\mockito-extensions\org.mockito.plugins.MockMaker to our Maven project with this content: Plain Text mock-maker-inline Use mock-inline Dependency The other, and probably better, option is adding mockito-inline dependency: XML <dependency> <groupId>org.mockito</groupId> <artifactId>mockito-inline</artifactId> <version>5.2.0</version> <scope>test</scope> </dependency> Note: this dependency already contains MockMaker resource mentioned above. Therefore, this option seems more convenient. Maven Warning No matter what version is used (see above), Maven build can produce these warnings: Plain Text WARNING: A Java agent has been loaded dynamically (<user_profile>\.m2\repository\net\bytebuddy\byte-buddy-agent\1.12.9\byte-buddy-agent-1.12.9.jar) WARNING: If a serviceability tool is in use, please run with -XX:+EnableDynamicAgentLoading to hide this warning WARNING: If a serviceability tool is not in use, please run with -Djdk.instrument.traceUsage for more information WARNING: Dynamic loading of agents will be disallowed by default in a future release OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended Mockito works correctly even with these warnings. It's probably caused by/depends on the used tool, JDK version, etc. Conclusion In this article, the mocking of static methods with the help of Mockito inline mock maker was covered. The article started with the basics of static mocking and then followed with a demonstration of when and verify usage (either with the method reference or the lambda expression). In the end, the setup of the Mockito inline maker was shown for different Mockito versions. The used source code can be found here. More

A Comprehensive Guide to Cloud Monitoring Tools: Ensuring Optimal Performance and Security

By Aditya Bhuyan

Introducing the Apache JMeter Docker Extension

By NaveenKumar Namachivayam

CORE

How to Monitor a Spring Boot App With Ostara

By Gunter Rotsaert

CORE

Use Amazon Bedrock and LangChain To Build an Application To Chat With Web Pages

Conversational interaction with large language model (LLM) based solutions (for example, a chatbot) is quite common. Although production grade LLMs are trained using a huge corpus of data, their knowledge base is inherently limited to the information present in the training data, and they may not possess real-time or the most up-to-date knowledge. Here is an example: This is perfectly acceptable. But, the real problem is "hallucination" wherein LLMs may generate inaccurate information, while sounding confident. Rather than having an open-ended conversation, it's good to narrow down the scope by providing additional context/information that's required to solve the problem or answer our questions. For example, instead of asking about Amazon Bedrock, one can provide a link to the documentation page (for instance General guidelines for Amazon Bedrock LLM users) and ask specific queries. The same can be done by Q&A over data in text or PDF documents. This can be achieved in many ways. The easiest one is to pass on the information to the LLM directly. But there are other popular techniques such as RAG (Retrieval Augmented Generation) that involve accessing data from external systems, Typically, this is done by combining Semantic search (with Vector databases). In this blog, we will explore the simple way (and leave the RAG technique for a future post). A framework like LangChain can simplify this for us since it provides abstractions to use the appropriate prompt (which can be customized), load data from sources (documents, web links) and inject it (as context) with the question/prompt. One of the previous blogs provided an introduction to using langchaingo with Amazon Bedrock, which is a fully managed service that makes base models from Amazon and third-party model providers (such as Anthropic, Cohere, and more) accessible through an API. It also walked you through how to extend langchaingo to work with the Anthropic Claude (v2) model in Amazon Bedrock. We will continue to build on that foundation and reuse the implementation. By the end, you will have a web application deployed to AWS App Runner that provides a fast, simple, and cost-effective way to deploy from source code or a container image directly to a scalable and secure web application. You can then use it to ask questions based on the content of a link/URL of your choice! The code is available on GitHub Application Overview The application is written in Go, but the concepts apply to any other language you might choose. As mentioned before, it uses langchaingo as the framework to interact with the Anthropic Claude (v2) model on Amazon Bedrock. The web app uses the Go embed package to serve the static file for the frontend part (HTML, JavaScript and CSS) from directly within the binary. To make sure that the contents of the link are included as context for the LLM, the applications uses the LoadStuffQA chain along with a prompt. You can refer to the application code here The chain is initialized in the init function: Go func init() { var err error region := os.Getenv("AWS_REGION") if region == "" { region = defaultRegion } llm, err = claude.New(region) chain = chains.LoadStuffQA(llm) } Note that the line llm, err = claude.New(region) comes from the langchaingo-amazon-bedrock-llm project that provides Amazon Bedrock extension for LangChain Go. The user can provide a link that serves as the source of information that they can ask questions. The content from the link will be represented as a Document. LangChain makes it easy to fetch data from various sources. In this case we fetch HTML content from a web URL (we will use AWS documentation as an example), but the same can be done for a text file or a PDF doc. The HTTP handler loadData invokes getDocsFromLink function that loads HTML data from the web link. The bulk of the work is done by this line - docs, err := documentloaders.NewHTML(resp.Body).Load(context.Background()) that combines NewHTML and Load functions to get the job done. Go func loadData(w http.ResponseWriter, r *http.Request) { //... details omitted } //... func getDocsFromLink(link string) ([]schema.Document, error) { resp, err := http.Get(link) defer resp.Body.Close() docs, err := documentloaders.NewHTML(resp.Body).Load(context.Background()) return docs, nil } Once contents of the user provided link are loaded, they can start asking questions. This is handled by the chat HTTP handler shown below (error handling and other parts omitted for brevity): Go func chat(w http.ResponseWriter, req *http.Request) { body, err := io.ReadAll(req.Body) chatMessage := string(body) answer, err := chains.Call(context.Background(), chain, map[string]any{ "input_documents": docs, "question": chatMessage, }, chains.WithMaxTokens(2048)) w.Write([]byte(answer["text"].(string))) } The user message (question) is fed as an input to the chains.Call function along with the content from the web URL/link. The LLM response is returned from the handler which is then rendered by the frontend to the user. Deploy the Application and Start Chatting... As a prerequisite, make sure you have Go, AWS CDK and Docker installed. You can easily deploy the entire solution with CDK. You can refer to the CDK code on GitHub Clone this GitHub repository, change to the right directory and start the deployment with cdk deploy. Shell git clone https://github.com/build-on-aws/amazon-bedrock-apprunner-chatterdox-webapp/ cd amazon-bedrock-apprunner-chatterdox-webapp/cdk export DOCKER_DEFAULT_PLATFORM=linux/amd64 cdk deploy This will start creating the AWS resources required for the application. You can keep track of the progress in the terminal or navigate to AWS console: CloudFormation > Stacks > ChatterdoxStack Once all the resources are created, you can try out the application. You should have: App Runner service - this is the web application App Runner Instance (IAM) role - this allows access to Amazon Bedrock Once complete, you should get a confirmation along with the values for the App Runner service endpoint. To access the application, enter the App Runner service endpoint URL in your web browser to navigate to the website. Start by entering a link to a page. For example, the NoSQL design for DynamoDB documentation page - AWS Documentation Once a valid link is provided (click Submit), the chat area will be enabled. You can now start asking questions about the page contents (enter a message in the text box and click Send). You can continue the chat conversation by asking additional questions. Conclusion Although this was a relatively simple application, it's evident how LangChain abstracted the complexity by providing an easy way to combine the following: Fetching data from the source (web link), Adding it as a context along with the prompt, and, Invoking the LLM Once the core functionality was ready, we were able to expose it as a user-facing web app on App Runner. This approach works quite well for single-page content or small documents. If you try to pass in a lot of content with the prompt, you will likely run into token limit constraints. To mitigate this, you can use other chains such as MapReduceDocuments, MapRerankDocuments etc. Other approach includes using RAG, which might be covered in a different blog. Until then, Happy Building!

By Abhishek Gupta

CORE

Accelerating Insights With Couchbase Columnar

The purpose of computing is insight, not numbers.- Richard Hamming Capella Columnar is an advanced real-time analytics database service from Couchbase, targeted for real-time data processing, offering SQL++ for processing JSON (semi-structured) data and more. This service enables data to be managed locally and streamed continuously from both relational and NoSQL databases, or simply process data on S3. The columns or fields of the source are directly mapped to a field in the JSON document at the destination automatically. This is really a zero-ETL operation. A key feature of this system is its ability to continuously stream data, making it immediately available for querying, thus ensuring near real-time data processing. At the heart of Columnar's architecture is a Massively Parallel Processing (MPP) engine, specifically designed to scale both capacity and performance. The SQL++ query processing system has been meticulously developed to leverage the scalable cloud architecture, incorporating efficient methodologies like distributed hash joins, a custom cost-based optimizer, and smart scanning of data on S3. The storage mechanism of Columnar is strategically engineered to boost performance while minimizing costs. It utilizes the first-in-the-NoSQL industry columnar storage for semi-structured data to reduce the storage footprint and improve query performance. Additionally, the system cleverly separates storage and compute, using S3 for persistence while caching data partitions on individual nodes for quick access. Designed for ease of use, the entire service can be deployed swiftly with just a few clicks. Users can interact with their data through a built-in workbench, or engage in natural language querying with the built-in GenAI tool iQ. The service allows for versatile output formats including JSON, tabular forms, or charts. Complementing its robust architecture, Columnar offers SDKs in all popular programming languages and native support for BI tools like Tableau and PowerBI, making it a comprehensive solution for modern data analytics needs. Enough details. Now, simply bring your data and accelerate your insights. Motivation For the Columnar Service As the holiday season approaches, a major global retailer faces a dire predicament that extends far beyond its corporate walls, directly impacting its end customers. Consider the frustration of shoppers who encounter out-of-stock notifications for their desired items, or who are bombarded with irrelevant promotions, all because the retailer is unable to quickly interpret and act upon the data pouring in from various sources. This is not simply a back-end data problem; it is a front-line customer experience crisis. Lost sales, diminished customer loyalty, and missed market opportunities are just the beginning. The retailer's inability to swiftly analyze and utilize data from its various databases and formats is costing not only in terms of revenue but also in terms of customer satisfaction and brand reputation. The holiday season, which should be a time of peak performance and customer engagement, is at risk of becoming a period of missed connections and lost potential. In the dynamic landscape of enterprise technology, where a mosaic of databases drives crucial operations from customer transactions to content management, a pressing challenge has surfaced. Enterprises navigate a complex web of systems: Oracle managing ERP, Couchbase powering e-commerce functionalities, MongoDB steering content management, with critical customer data often tucked away in S3 files. This diverse data environment, although instrumental in fostering microservice autonomy and scalability, uncovers a daunting challenge: the real-time identification and resolution of bottlenecks in the customer journey. This issue is emblematic of a broader spectrum of hurdles that modern enterprises encounter in their quest to orchestrate an effective "observe, orient, decide, and act" (OODA) loop, a cornerstone for operational efficiency and competitiveness today. Gaining actionable insights requires harnessing a vast expanse of data. The key lies not just in amassing this data but in rapidly analyzing and effectively visualizing it in its myriad forms to unearth those critical insights. This is precisely the objective at the heart of Couchbase's Capella Columnar service. Designed to bridge the gap between extensive data collection and insightful analysis, Details of Couchbase Columnar Service Since a picture is worth a thousand words, the picture below should answer all your questions! :-) For those who are even more curious, read on... It's a sophisticated service with significant features to make it easy to analyze. It's difficult to provide all of the details in a single article. Here's an overview! Data Objects Columnar service allows you to create databases that can have scopes. A scope is a logical entity that can have one or more collections and their indexes. Each collection can be simply a "standalone collection" whose data is manipulated by SQL++ DML statements: INSERT, UPSERT, DELETE, SELECT. The collection can also point to a collection or a table in an external database instance. These external databases can be Couchbase, MongoDB, DynamoDB, Cassandra, or even RDBMS such as MySQL. With a few clicks to provide secure access to data, you can set up a collection that has a copy of the remote data. Not only does it copy the data but it then automatically sets up the infrastructures to continuously stream changes at a remote database to your collection in a matter of milliseconds to seconds. The data is immediately available for analysis. The next type of collection is the "external data set" that simply points to a file or a structure on S3. The data can be in JSON, Parquet, CSV, and TSV now with more formats to be supported in the future. The support for S3 is designed to handle very large data processing by identifying and reading only the data required by the query. Data Sources Currently, we support the following non-local data sources: Couchbase on-prem Couchbase Capella S3 data in CSV, TSV, JSON, and Parquet. More format support is on the way. MySQL MongoDB and MongoDB Atlas DynamoDB Cassandra support is in the way Architecture The core database, built on a Massively Parallel Processing (MPP) engine, uses JSON as its super data model to support the flexibility required by modern data. This system supports SQL++ query language that supports the standard select, join, and project operations as well as advanced nesting, unnesting, windowing, rollup, and cubing operations declaratively. It's equipped with indexes on both scalar and array data types. Queries on these are efficiently planned by a specialized cost-based optimizer, tailored for JSON and MPP architecture. SQL++, enhanced from SQL, is adept at handling both flat data and JSON and its data types. This capability positions SQL++ as an effective query language for managing heterogeneous data, mirroring the role of JSON as the foundational data model. For a more comprehensive understanding of this technology and the various facets of the core database server, accompanying papers offer in-depth analyses and insights. Here are papers that give you a deeper look into technology and many aspects of the core database server. Architecture Overview: Couchbase Analytics: NoETL for Scalable NoSQL Data Analysis Array Indexes: On Multi-Valued Indexing in AsterixDB SQL++ For SQL Users: A Tutorial Theory behind SQL++: The SQL++ Query Language: Configurable, Unifying, and Semi-structured Storage We now focus on storage within Capella Columnar, specifically its columnar storage model. The combination of the JSON document model with columnar storage is highly advantageous from a user perspective, though it presents significant implementation challenges due to the flexible schema inherent in JSON. Typically, a JSON document stores considerably more data than a standard row in a traditional RDBMS. This difference becomes particularly relevant during data analysis, which often involves just a few columns per query, leading to substantial I/O savings. When implemented effectively, columnar storage not only offers these I/O benefits but also brings additional advantages in terms of data compression. So, the columnar storage improves analysis performance and reduces TCO simultaneously. The columnar storage utilized in Capella Columnar is the culmination of research conducted at the University of California at Irvine and subsequently refined and hardened by Couchbase. Again, detailed information on columnar storage is in the paper "Columnar Formats for Schemaless LSM-based Document Stores." iQ for Columnar iQ is a natural language interface to interact with your data by generating the SQL++ query for your ask. For Columnar, we've improved the query generation quality for joins. And finally, with one single click, you can go from your natural language question to a chart. Click here to play the GIF.

By Keshav Murthy

CORE

A Maven Story

Back with a fresh blog on Maven – no saga, just a simple update on an old project of mine. It's like catching up with an old buddy and sharing what's new. In this blog, we'll dive into the world of Maven, exploring the tweaks and improvements I've made to an old favorite. Whether you're a Maven pro or just getting started, let's take a stroll through My Maven adventure together! TL;DR The Challenge: Figuring out how to upgrade an application server in a project with lots of parts using Ant. The Solution: Use Maven's Multi-Module plan and a Bill Of Materials (BOM) to control versions in one place, making upgrades and managing parts easier. The Journey: Facing issues like circular dependencies and copying projects, Maven's tools, and multi-module system helped a lot. Key Takeaways: Sharing practical lessons learned during the move, showing how Maven made the project better and ready for more improvements. Once Upon a Time in Engineering In a tech adventure, I have been tasked to upgrade the Application Server for a product used on servers like Glassfish or WebLogic. The goal? Boost security and prepare for future JDK upgrades. The product included 20+ EJB projects and 20+ war projects packed into an EAR. These EJB(*-ejb.jar) and WEB (*.war projects) were Ant based Java projects. The Ant Challenge In this world of Ant projects, a big problem arose—figuring out and upgrading application server-related dependencies. The current setup used two projects holding all kinds of dependencies, making it hard to know which project used what. Updating during server upgrades turned into a manual, error-prone task. The Maven Journey To tackle these issues, I aimed to simplify future upgrades and centralize dependency control. Although Ant IVY seemed helpful, I worried about managing dependencies and the messy project structure. Maven and Gradle were options, but Maven gaining the upper hand due to its robust tools and plugins for application servers. Taking cues from open-source projects and the structure of Spring projects, the Maven Multi-module with Bill Of Materials (BOM) emerged as a compelling choice. Choosing the Path To make it easier to understand, think of the picture below as the old way we organized projects. Module_A projects in Ant are compiled into four binaries. However, this contradicts the Maven policy of having One binary per POM by maintaining a pom.xml for each Ant project. Here comes the pivotal question: Could I seamlessly integrate Maven into the existing Ant project structure? Laying New Road The answer lay in creating a parent project with a POM serving as a BOM. This approach offered a refined project structure, aligning with Maven's directory layout standards. The final setup turned into a Maven Multi-Module project, with /Application/pom.xml controlling versions for all modules. The application_ear project wrapped EJB and WAR files into a deployable EAR, while BuildModules made an executable application server zip. In this context, /Module_A itself functions as a Maven module with /Application as its parent. Moreover, for each binary of Module_A created in the old project structure, I established a Maven module with /Module_A as the parent. This design mirrors an ideal Maven multi-module project structure for our project. The *_ejb project compiles EJB classes into module_a_ejb.jar. The *_web project assembles WAR classes into module_a_web.war. Additionally, *_interface projects bundle Interfaces and DTOs for inter-module dependencies into module_a_interface.jar. I'll delve into the interesting reason behind this *_interface packaging in a later section of this blog. Deep Dive As previously mentioned, I've established a primary Maven project called /Application, listing all the subsidiary domain projects beneath it. Concerning version control, the /Application/pom.xml serves as a Bill-Of-Materials, encompassing dependency and plugin management for centralized control. The XML snippet below showcases version declarations for log4j-1.2-api and Gson. XML <properties> <version.log4j.1.2.api>2.19.0</version.log4j.1.2.api> <version.gson>2.8.6</version.gson> </properties> <dependencyManagement> <dependency> <groupId>org.apache.logging.log4j</groupId> <artifactId>log4j-1.2-api</artifactId> <version>${version.log4j.1.2.api}</version> </dependency> <dependency> <groupId>com.google.code.gson</groupId> <artifactId>gson</artifactId> <version>${version.gson}</version> </dependency> </dependencyManagement> Consequently, all sub-modules inheriting or using these dependencies are relieved from specifying version information, as demonstrated in the subsequent XML snippet: XML <dependency> <groupId>org.apache.logging.log4j</groupId> <artifactId>log4j-1.2-api</artifactId> <scope>compile</scope> </dependency> <dependency> <groupId>com.google.code.gson</groupId> <artifactId>gson</artifactId> <scope>compile</scope> </dependency> Moreover, the concept of inheritance plays a role where Maven POMs inherit values from a parent POM. This includes Maven plugins defined at the parent level, providing default implementations for sub-module plugins, as exemplified below: XML <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-jar-plugin</artifactId> <version>${version.maven.jar.plugin}</version> <configuration> <archive> <index>false</index> <manifestEntries> <Built-By>${user.name}</Built-By> <Build-Date>${maven.build.timestamp}</Build-Date> <Build-Jdk>${java.vendor}</Build-Jdk> </manifestEntries> <manifest> <addClasspath>false</addClasspath> </manifest> </archive> </configuration> </plugin> To handle internal or in-house dependencies, we've established an internal Maven repository akin to JFrog Artifactory within our intranet network. A dedicated repository has been crafted for our product dependencies, and this repository has been integrated for effective dependency resolution. XML <repositories> <repository> <name>Internal Dev</name> <id>maven-dev</id> <url>${maven.dev.repo.url}</url> <layout>default</layout> <releases> <updatePolicy>never</updatePolicy> </releases> <snapshots> <updatePolicy>always</updatePolicy> </snapshots> </repository> </repositories> To ensure uniformity in Java distribution and version during compilation, we've encountered instances of unintentional compilation with different JDK versions in the Ant build model. Addressing this concern, I resolved the issue using the maven enforcer plugin, This plugin allows us to enforce rules on JDK version and distribution, as illustrated in the example below: XML <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-enforcer-plugin</artifactId> <version>${version.maven.enforcer.plugin}</version> <executions> <execution> <id>enforce-java</id> <goals> <goal>enforce</goal> </goals> <configuration> <rules> <requireJavaVendor> <excludes> <exclude>${oracle.jdk}</exclude> </excludes> <includes> <include>${eclipse.temurin.jdk}</include> </includes> </requireJavaVendor> </rules> </configuration> </execution> </executions> </plugin> </plugins> </build> Facing Challenges Throughout our smooth migration process, we encountered a few hurdles: 1. 160+ Project Creation: With over 40 main modules totaling 160+ projects and submodules, replicating the project structure across all with default configurations and files proved time-consuming. To streamline this, I utilized the Maven project templating kit Archetype. This tool facilitates the creation of templated project structures, including resource files and default classes for submodules. 2. Circular Dependencies: Given the Ant-based nature of the old projects and the absence of a strict component dependency model, circular dependencies emerged as a challenge during project compilation in multiple areas. Maven does not permit circular dependencies. To address this, I introduced *-ejb, *-web, and *-interfaces projects under submodules, segregating Java classes by concern. The INTERFACE projects act as saviors, resolving circular dependencies by holding only DTOs and EJB Interfaces without inter-project dependencies. Other module EJB/WEB projects now reference either their self-module or other modules' *-interfaces as dependencies. 3. Distribution Packaging: Creating a product distribution that bundles *.ear with the application server into a zip file was essential. Since the complete product build script is based on Ant, I decided to employ the maven assembly plugin for packaging the distribution zip file. Initially, I considered placing the assembly scripts in the parent project /Application, assuming that building the parent project would cover all submodules. However, this approach led to inherited assembly behavior across submodules, causing confusion about distribution dependencies and configuration. Despite attempts to restrict inheritance to submodules, the solution proved ineffective. Referring to the assembly plugin guide, I then crafted a distribution project named BuildModules. This project generates the zip file alongside the EAR and application server, with dependencies over the EAR modules, which, in turn, have transitive dependencies over the submodule binaries (*-ejb.jar and *.war). The Grand Finale Despite surpassing the estimated timeline, the project's ultimate results were commendable. Subsequent minor upgrades to the Application Server were effortlessly accomplished with a mere Maven property change. Presently, the OWASP dependency-check-maven conducts more effective scans of dependencies for any known published vulnerabilities. Upgrading the project's dependencies to align with compatible JDK versions or distributions proved to be a straightforward and uncomplicated process. In the end, it's a well-rounded success story. Moving forward, we're delving into Maven improvements and developing custom plugins for the product in the next phase. Thank you for joining in on this journey. I look forward to sharing more insights in the upcoming blog.

By Thirumalaivasan Rajasekaran

Beyond Jenkins: The Power of Cloud-Native AWS CodePipeline for Deployments

Tools and platforms form the backbone of seamless software delivery in the ever-evolving world of Continuous Integration and Continuous Deployment (CI/CD). For years, Jenkins has been the stalwart, powering countless deployment pipelines and standing as the go-to solution for many DevOps professionals. But as the tech landscape shifts towards cloud-native solutions, AWS CodePipeline emerges as a formidable contender. Offering deep integration with the expansive AWS ecosystem and the agility of a cloud-based platform, CodePipeline is redefining the standards of modern deployment processes. This article dives into the transformative power of AWS CodePipeline, exploring its advantages over Jenkins and showing why many are switching to this cloud-native tool. Brief Background About CodePipeline and Jenkins At its core, AWS CodePipeline is Amazon Web Services' cloud-native continuous integration and continuous delivery service, allowing users to automate the build, test, and deployment phases of their release process. Tailored to the vast AWS ecosystem, CodePipeline leverages other AWS services, making it a seamless choice for teams already integrated with AWS cloud infrastructure. It promises scalability, maintenance ease, and enhanced security, characteristics inherent to many managed AWS services. On the other side of the spectrum is Jenkins – an open-source automation server with a storied history. Known for its flexibility, Jenkins has garnered immense popularity thanks to its extensive plugin system. It's a tool that has grown with the CI/CD movement, evolving from a humble continuous integration tool to a comprehensive automation platform that can handle everything from build to deployment and more. Together, these two tools represent two distinct eras and philosophies in the CI/CD domain. Advantages of AWS CodePipeline Over Jenkins 1. Integration with AWS Services AWS CodePipeline: Offers a native, out-of-the-box integration with a plethora of AWS services, such as Lambda, EC2, S3, and CloudFormation. This facilitates smooth, cohesive workflows, especially for organizations already using AWS infrastructure. Jenkins: While integration with cloud services is possible, it usually requires third-party plugins and additional setup, potentially introducing more points of failure or compatibility issues. 2. Scalability AWS CodePipeline: Being a part of the AWS suite, it natively scales according to the demands of the deployment pipeline. There's no need for manual intervention, ensuring consistent performance even during peak loads. Jenkins: Scaling requires manual adjustments, such as adding agent nodes or reallocating resources, which can be both time-consuming and resource-intensive. 3. Maintenance AWS CodePipeline: As a managed service, AWS handles all updates, patches, and backups. This ensures that the latest features and security patches are always in place without user intervention. Jenkins: Requires periodic manual updates, backups, and patching. Additionally, plugins can introduce compatibility issues or security vulnerabilities, demanding regular monitoring and adjustments. 4. Security AWS CodePipeline: One of the key benefits of AWS's comprehensive security model. Features like IAM roles, secret management with AWS Secrets Manager, and fine-grained access controls ensure robust security standards. Jenkins: Achieving a similar security level necessitates additional configurations, plugins, and tools, which can sometimes introduce more vulnerabilities or complexities. 5. Pricing and Long-Term Value AWS CodePipeline: Operates on a pay-as-you-go model, ensuring you only pay for what you use. This can be cost-effective, especially for variable workloads. Jenkins: While the software itself is open-source, maintaining a Jenkins infrastructure (servers, electricity, backups, etc.) incurs steady costs, which can add up in the long run, especially for larger setups. When Might Jenkins Be a Better Choice? Extensive Customization Needs With its rich plugin ecosystem, Jenkins provides a wide variety of customization options. For unique CI/CD workflows or specialized integration needs, Jenkins' vast array of plugins can be invaluable, including integration with non-AWS services. On-Premise Solutions Organizations with stringent data residency or regulatory requirements might prefer on-premise solutions. Jenkins offers the flexibility to be hosted on local servers, providing complete control over data and processes. Existing Infrastructure and Expertise Organizations with an established Jenkins infrastructure and a team well-versed in its intricacies might find transitioning to another tool costly and time-consuming. The learning curve associated with a new platform and migration efforts can be daunting. The team needs to weigh in on the transition along with other items in their roadmap. Final Takeaways In the ever-evolving world of CI/CD, selecting the right tool can be the difference between seamless deployments and daunting processes. Both AWS CodePipeline and Jenkins have carved out their specific roles in this space, yet as the industry shifts more towards cloud-native solutions, AWS CodePipeline indeed emerges at the forefront. With its seamless integration within the AWS ecosystem, innate scalability, and reduced maintenance overhead, it represents the future-facing approach to CI/CD. While Jenkins has served many organizations admirably and offers vast customization, the modern tech landscape is ushering in a preference for streamlined, cloud-centric solutions like AWS CodePipeline. The path from development to production is critical, and while the choice of tools will vary based on organizational needs, AWS CodePipeline's advantages are undeniably compelling for those looking toward a cloud-first future. As we navigate the challenges and opportunities of modern software delivery, AWS CodePipeline offers a promising solution that is more efficient, scalable, secure, and worth considering.

By Prithvish Kovelamudi

Comparative Analysis of Open Source Cluster Management Systems: Kubernetes vs. Apache Mesos

Cluster management systems are critical software solutions that enable the efficient allocation and utilization of computing resources in a network of interconnected machines. No doubt, they play a vital role in modern computing by ensuring scalability, high availability, and effective resource management, making them essential for running complex applications, managing data centers, and further multiplying the power of distributed computing. As reported by National Grid ESO, data centers, despite all advancements, still account for a significant 1% of global electricity consumption, and that is where Cluster Management Systems might play a crucial role in enhancing energy efficiency. Before we dive into the details, it's important to note that this article is not about declaring one system as the "better" choice. Instead, we're starting a journey to compare and contrast two prominent open-source cluster management systems, Kubernetes and Apache Mesos, for they have quite different approaches. We'll shine a light on their unique features, strengths, and weaknesses, helping one make informed decisions based on one’s specific needs. So, whether you're a seasoned IT professional looking to fine-tune your cluster management strategy or someone new to the world of distributed systems, join us on this route as we dissect and explore the fascinating realm of Kubernetes and Apache Mesos. It's all about understanding the nuances and making the right choice for your next big project. Why Them Specifically? The Reason Behind the Comparison Comparing Kubernetes and Mesos is a strategic choice born out of their prominence in the world of cluster management systems. These two open-source solutions have earned substantial attention, boasting large user communities, diverse use cases, and robust ecosystems of tools and extensions. While there are indeed other cluster management tools available, such as Docker Swarm or Nomad, Kubernetes, and Mesos often appear as top contenders in discussions about large-scale orchestration and resource management. This comparison is a starting point for understanding the fundamental approaches and philosophies behind different cluster management systems. Background Information: Kubernetes Kubernetes was born at Google. It evolved from their internal Borg system and its offspring, the experimental cluster manager Omega. Google open-sourced Kubernetes in 2014, and since then, it's grown into a crushing force with a thriving open-source community. As reported in the Kubernetes Companies table dashboard, among its contributors are such eminent tech companies as Google itself (128 contributions in the last six months), Red Hat (109), Microsoft (55), and others. Key Features and Concepts Kubernetes can be viewed as the core part, providing storage and a suite of APIs for constructing distributed systems, complemented by a robust set of built-in objects and controllers like the "batteries-included" package. Some of its prominent features include: Pods: Pods are the smallest units of work in Kubernetes, grouping one or more containers together. Services: They assist the applications in communication with each other, whether they're in the same pod or scattered across the cluster. Replication Controllers: These keep the applications running smoothly by making sure one has the right number of copies (replicas) running. Load Balancing: Kubernetes can distribute traffic evenly to the application replicas; it ensures the users get a smooth experience. Introduction to Apache Mesos The journey of Apache Mesos started back at the University of California, Berkeley, and it was open-sourced in 2010. Initially, it was a research project conducted by PhD student Benjamin Hindman. In the subsequent, Hindman collaborated a lot with John Wilkes, one of the authors of the Omega mentioned above: they cooperated extensively on the design of Apache Mesos and Omega, though their respective approaches eventually took distinct paths in the realm of cluster management. Now, Apache Mesos is a robust framework used by companies like Twitter (now X) and Airbnb. Key Features and Concepts Mesos is not just about containers but rather about managing resources like CPU and memory across your entire data center. As stated in the whitepaper by its creators, Mesos allocates resources in a fine-grained manner, letting frameworks achieve data proximity by alternating reads from data stored on each machine. Some of its features are: Resource Allocation: As mentioned above, Mesos can divide the data center's resources, allocating them to applications dynamically. Frameworks: Think of these as specialized managers for different types of workloads, like running Spark for big data or a web server for one’s website. Fault Tolerance: Mesos is known for its resilience, managing hardware failures using Zookeeper for fault-resistant coordination and utilizing sharding techniques that synchronize with host agents post-leader failure. Multi-Tenancy: It is able to run different workloads on the same cluster without interruption. In summary, we have Kubernetes, the sophisticated container orchestrator, and Mesos, the master of resource allocation. These introductions set the stage for the deep examination of their worlds. Comparison Architecture and Design Kubernetes and Mesos come at cluster management from different angles. On the one hand, Mesos master extends offers to application schedulers (known as "frameworks"), which they can choose to accept or decline; on the other hand, Kubernetes enables clients (be it controllers or even via CLI) to submit a resource request (in the form of a Pod) to a specific scheduler that satisfies those requests. Scalability and Performance Kubernetes has a mastery of scaling applications up or down. Its auto-scaling features allow for adapting to changing workloads seamlessly. Kubernetes also has built-in load balancing, which helps distribute traffic smoothly to keep the apps performing. Mesos, with its fine-grained resource allocation, showcases the great performance. It can allocate resources with great precision, making it suitable for diverse workloads. It's the most suitable for resource allocation and ensuring efficient use of the cluster's resources. Ecosystem and Community Kubernetes has a massive and vibrant community. The ecosystem is vast, with tools like Helm for packaging apps, Prometheus for monitoring, and Grafana for visualization. Other than that, Kubernetes has gained extensive support from major cloud providers such as Google Cloud's GKE, Amazon's EKS, and Microsoft Azure's AKS. Mesos has a smaller community but still has its share of frameworks and libraries. Apache Spark and Hadoop are some famous frameworks that call Mesos home. While Kubernetes sees broader managed service support, Mesos also receives backing from various cloud providers, including Microsoft and Oracle, which announced support for it on their cloud platforms, Azure and Oracle Container Cloud Service, respectively. Ease of Use and Learning Curve Kubernetes has significantly advanced in usability, yet it may present complexities for those unfamiliar with its ecosystem. It necessitates some learning, particularly concerning YAML files and their distinct terminology. Mesos, on the other hand, offers a more straightforward initiation for those acquainted with Linux. Nonetheless, constructing custom frameworks presents its own set of challenges and demands diligence. Fault Tolerance and High Availability Kubernetes has robust fault tolerance, as it is built on top of etcd, a distributed, reliable key-value store. If a pod goes down, Kubernetes resurrects it. Mesos handles failures similarly as it relies on Zookeeper with similar use cases as Kubernetes, but fault tolerance often depends on the frameworks you're using. Security Kubernetes offers strong security features with role-based access control, network policies, and pod security policies. Mesos has security measures like framework isolation and authentication. It ensures that your frameworks can't trample over each other. What Might Be the Choice? The choice between Kubernetes and Apache Mesos depends on various factors, including the specific use case, requirements, and organizational context. There is no one universal answer, as both cluster management systems have their strengths and weaknesses. Here are some considerations to help one make an informed decision: Choose Kubernetes if the most important is: Container orchestration Community and ecosystem Ease of use Standardization Choose Apache Mesos if you value: Resource flexibility Multi-Tenancy Advanced use cases Customization Legacy Integration Ultimately, the choice depends on the specific requirements, existing infrastructure, and the expertise of the team. In some cases, organizations may even choose to use both Kubernetes and Mesos within their environments, each serving a distinct purpose. It's crucial to evaluate both systems thoroughly and consider how well they align with the long-term goals and technical constraints before making a decision — and hopefully, this article was able to assist you in that.

By Nikita Vetoshkin

Streamlining Data Lake ETL With Apache NiFi: A Practical Tutorial

In the world of data-driven decision-making, ETL (Extract, Transform, Load) processes play a pivotal role. The effective management and transformation of data are essential to ensure that businesses can make informed choices based on accurate and relevant information. Data lakes have emerged as a powerful way to store and analyze massive amounts of data, and Apache NiFi is a robust tool for streamlining ETL processes in a data lake environment. Understanding Data Lake ETL Before diving into Apache NiFi, let's clarify what ETL means in the context of data lakes. Data Lakes: What Are They? Data lakes are repositories for storing vast amounts of structured and unstructured data. Unlike traditional databases, data lakes do not require data to be pre-structured before it's stored. This makes data lakes suitable for storing raw, diverse data, which can then be processed and analyzed as needed. ETL in Data Lakes ETL stands for Extract, Transform, Load. It's a process that involves: Extracting data from various sources Transforming the data to make it suitable for analysis Loading the transformed data into the data lake ETL is crucial for ensuring that the data in the data lake is clean, consistent, and ready for analysis. Challenges in Data Lake ETL Handling ETL processes in a data lake can be challenging for several reasons: Data variety: Data lakes store different data types, including structured and unstructured data, which must be transformed and processed differently. Data volume: Data lakes handle vast amounts of data, often in the petabyte range, making efficient data movement and processing critical. Data velocity: Data is continually ingested into the data lake, and ETL processes must keep up with this fast data flow. Data quality: Ensuring data quality is essential, as poor-quality data can lead to inaccurate insights. Introduction to Apache NiFi Apache NiFi is an open-source data integration tool that provides a powerful and user-friendly way to design data flows. It is well-suited for ETL processes in data lakes due to its flexibility, scalability, and data provenance capabilities. Key Features of Apache NiFi User-friendly interface: NiFi offers a drag-and-drop interface, making it accessible to both technical and non-technical users. Data provenance: NiFi tracks the data's journey from source to destination, allowing you to trace data lineage and monitor data quality. Scalability: NiFi can scale horizontally to handle large data volumes and is designed for high availability. Why Choose Apache NiFi for Data Lake ETL? NiFi's flexibility and versatility make it an excellent choice for data lake ETL: It supports various data sources and destinations, including Hadoop HDFS, AWS S3, Azure Data Lake Store, and many others. Its data transformation capabilities enable you to process data in real-time. Built-in security features ensure that data is protected during the ETL process. Setting up Apache NiFi Let's get started with setting up Apache NiFi for your data lake ETL. 1. Installation You can download Apache NiFi from the official website. Follow the installation instructions for your specific environment, whether it's on-premises or in the cloud. Be sure to meet the system requirements and install any necessary dependencies. 2. Configuration After installation, you'll need to configure NiFi to suit your needs. This involves defining data sources, configuring processors, and setting up connections between components. The NiFi interface is intuitive and user-friendly. You'll create a data flow by dragging processors onto the canvas and connecting them to define the flow of data. Building ETL Workflows With NiFi Now, let's explore how to build ETL workflows using Apache NiFi. Creating Data Pipelines To create an ETL workflow in NiFi, follow these steps: Define data sources and destinations. Add processors to perform data extraction, transformation, and loading. Connect processors to define the flow of data. For instance, you can set up a data pipeline that extracts data from an FTP server, transforms it into a structured format, and loads it into your data lake. Data Transformation NiFi provides various processors for data transformation, including: ConvertRecord: Convert data from one format to another. SplitText: Split text data into individual records. MergeContent: Merge multiple records into a single data flow file. By configuring these processors, you can tailor your data transformation to meet your specific ETL requirements. Data Ingestion and Loading NiFi supports a wide range of data destinations. You can easily configure processors to send data to Hadoop HDFS, cloud storage services like AWS S3, databases, or other data lake storage platforms. This flexibility allows you to adapt your ETL processes to your data lake's requirements. Data Lake Integration One of the strengths of Apache NiFi is its seamless integration with various data lake platforms. Hadoop HDFS Integration To integrate NiFi with Hadoop HDFS: Configure the PutHDFS processor to define the destination directory and set up Hadoop connection properties. You can also use the ListHDFS processor to retrieve file listings from HDFS. AWS S3 Integration For integration with AWS S3: Configure the PutS3Object processor to specify the S3 bucket, key, and access credentials. The GetS3Object processor can be used to retrieve objects from S3. Azure Data Lake Store Integration To connect NiFi to Azure Data Lake Store: Configure the PutAzureDataLakeStore processor with your Azure Data Lake Store credentials and target path. Use the FetchAzureDataLakeStore processor to retrieve data from the data lake. This flexibility allows you to seamlessly integrate NiFi with your chosen data lake platform. Monitoring and Management Apache NiFi provides tools for monitoring and managing ETL processes. Data Provenance Data provenance in NiFi is a powerful feature that allows you to track the data's journey. It records all actions on data flow files, helping you trace the origins of your data and identify any issues in your ETL pipeline. Logging and Alerts NiFi offers extensive logging capabilities, which can be essential for troubleshooting. You can set up alerts and notifications to be informed of any errors or issues in your ETL processes. Performance Optimization Optimizing ETL performance is critical for data lake operations. Load Balancing For high data volumes, consider setting up load balancing between multiple NiFi instances. This helps distribute the workload and ensures better performance and fault tolerance. Clustering NiFi can be configured in a clustered setup, providing scalability and high availability. In a cluster, NiFi instances work together to manage data flows and provide redundancy. Resource Allocation Properly allocate system resources (CPU, memory, and network bandwidth) to ensure that NiFi can efficiently process data. Resource allocation ensures that your ETL workflows run smoothly and meet the performance demands of your data lake. Security and Data Governance In a data lake environment, security and data governance are paramount. Apache NiFi offers features to ensure data protection and compliance. 1. Data Encryption NiFi supports data encryption both at rest and in transit. You can configure SSL/TLS to secure data while it's being transferred between components, ensuring data confidentiality and integrity. 2. Authentication and Authorization NiFi allows you to set up user authentication and authorization, ensuring that only authorized users can access and modify ETL processes. This is crucial for maintaining data security and compliance with data governance regulations. 3. Data Lineage and Auditing With NiFi's data provenance and auditing features, you can track every action taken on your data. This audit trail helps in compliance with data governance requirements and provides transparency in data management. Real-World Use Cases To illustrate the practical application of Apache NiFi in streamlining data lake ETL, let's explore a couple of real-world use cases. Use Case 1: E-commerce Data Processing Imagine an e-commerce company that collects massive amounts of customer data, including browsing history, purchase records, and customer reviews. This data needs to be ingested into a data lake, transformed into a structured format, and loaded for analysis. By implementing Apache NiFi, the company can create ETL pipelines that extract data from various sources, transform it to meet analysis requirements and load it into their data lake. NiFi's real-time processing capabilities ensure that the latest data is available for analysis. Use Case 2: Financial Services A financial services institution deals with a constant stream of financial transactions, customer records, and market data. It's crucial to efficiently process this data and make it available for risk assessment and compliance reporting. Using Apache NiFi, the institution can create ETL workflows that continuously ingest and process this data. Data is transformed, enriched, and loaded into the data lake, providing real-time insights and ensuring compliance with financial regulations. In both use cases, Apache NiFi's flexibility, scalability, and data lineage features make it an ideal tool for handling complex ETL processes in data lake environments. Conclusion Streamlining ETL processes in a data lake is essential for organizations aiming to leverage their data effectively. Apache NiFi provides a user-friendly, powerful solution for designing and managing data flows, making it a valuable tool for data engineers and analysts. In this practical tutorial, we've covered the fundamentals of data lake ETL, introduced Apache NiFi, and explored its features and benefits. You've learned how to set up NiFi, create ETL workflows, integrate it with data lake platforms, monitor and manage ETL processes, optimize performance, and ensure data security and governance. By following the steps outlined in this tutorial, you can harness the capabilities of Apache NiFi to streamline your data lake ETL processes, making your data more accessible, reliable, and valuable for data-driven decision-making. Whether you're working with a small-scale data lake or managing petabytes of data, Apache NiFi can help you meet the challenges of data lake ETL with confidence.

By Lav Kumar

Finding the Right Database Solution: A Comprehensive Comparison of AWS RDS MySQL and Aurora MySQL

Choosing the right database solution is an essential factor that could significantly influence your application’s overall performance. This article aims to provide a comprehensive comparison between AWS RDS MySQL and Aurora MySQL, two powerful database solutions offered by Amazon Web Services (AWS). I will delve into the specifics of their architecture, performance, data replication capabilities, security measures, cost efficiency, ease of use, integration capabilities, and support resources. By the end of this guide, you will be equipped with all the necessary information to make an informed decision about the most suitable database solution for your specific needs. AWS RDS MySQL and Aurora MySQL are both managed database services offered by Amazon Web Services. AWS RDS MySQL is a relational database service that provides cost-efficient and resizable capacity while automating time-consuming administration tasks. On the other hand, Aurora MySQL is a MySQL-compatible relational database engine that offers superior performance akin to high-end commercial databases at a fraction of the cost. The right database solution not only ensures efficient data management but also supports your applications' performance and scalability requirements. It can help you avoid potential downtime, enhance application responsiveness, and ensure data security and compliance. Thus, understanding the nuances of AWS RDS MySQL and Aurora MySQL becomes crucial in determining the best fit for your particular scenario. Architecture and Performance AWS RDS MySQL uses a traditional monolithic architecture where the database exists on a single server or multiple servers working as one unit. This setup allows it to deliver a very fast, multi-threaded, and robust SQL database server, making it an ideal choice for mission-critical and heavy-load production systems. However, its architecture might have limitations when dealing with extremely high workloads. Unlike RDS MySQL, Aurora MySQL employs a distributed, fault-tolerant, self-healing storage system that auto-scales up to 64TB per database instance. This architecture enables Aurora MySQL to offer up to five times better performance than MySQL, making it a top choice for demanding applications that require high throughput and low latency. When we compare AWS RDS MySQL and Aurora MySQL in terms of performance and scalability, Aurora tends to outshine RDS MySQL. While RDS MySQL offers robust performance for a wide range of applications, Aurora MySQL's distributed architecture allows it to handle higher workloads, offering superior performance and scalability. However, the choice between the two will heavily depend on your specific workload and performance requirements. Data Replication and Availability AWS RDS MySQL supports data replication through its Read Replicas feature, allowing you to create up to five copies of your database. This process aids in enhancing the database's availability and durability. However, compared to Aurora MySQL, RDS MySQL's replication process is relatively slower. Aurora MySQL takes data replication a notch higher by allowing you to provision up to 15 replicas, and it performs replication in milliseconds. This quick replication process, coupled with automatic failover, mitigates data loss risks and ensures higher data availability. In terms of data availability and replication speed, Aurora MySQL has the upper hand over RDS MySQL. The ability to provision up to 15 replicas and its lightning-fast replication process make Aurora MySQL more resilient and reliable, especially for applications that demand high data availability. Security and Compliance AWS RDS MySQL offers robust security features, including network isolation using Amazon VPC, encryption at rest and in transit, IAM integration for access control, and automated patches and updates. It also complies with several key industry standards, providing a secure environment for your data. Just like RDS MySQL, Aurora MySQL also provides robust security features, including encryption at rest and in transit, network isolation using Amazon VPC, and IAM integration. Additionally, Aurora MySQL includes advanced features like database activity streams for real-time monitoring of the database, further enhancing its security posture. Both RDS MySQL and Aurora MySQL offer strong security features, ensuring that your data is protected against potential threats. However, Aurora MySQL's additional capabilities, like real-time database activity streams, give it a slight edge over RDS MySQL when it comes to security. Cost Efficiency AWS RDS MySQL follows a pay-as-you-go pricing model. The costs are based on the resources consumed, such as compute instances, storage, and data transfer. This flexible pricing structure can be cost-effective, especially for small to medium-sized workloads. Just like RDS MySQL, Aurora MySQL also follows a pay-as-you-go pricing model, with charges based on the resources used. However, considering its superior performance and scalability features, Aurora MySQL delivers similar performance to high-end commercial databases at almost one-tenth the cost. While both RDS MySQL and Aurora MySQL offer cost-effective solutions, the choice between the two should center around your specific requirements. If you require a database for small to medium-sized workloads, RDS MySQL could be your cost-effective choice. However, if you're dealing with high-volume workloads and need superior performance and scalability, Aurora MySQL's high-end features might justify its higher costs. Ease of Use and Management AWS RDS MySQL offers automated backups, software patching, automatic failover, and recovery mechanisms, which significantly reduce the administrative burden. It also allows easy scaling of compute resources and storage capacity to meet the demands of your application. Aurora MySQL also provides a fully managed service that automates time-consuming tasks such as hardware provisioning, database setup, patching, and backups. Furthermore, it allows on-the-fly modifications to the instance type or storage, providing flexibility in managing your database operations. Both RDS MySQL and Aurora MySQL provide a fully managed experience, simplifying database management. However, Aurora MySQL's ability to make on-the-fly adjustments to instance types and storage adds an extra layer of flexibility, making it slightly more user-friendly in terms of management. Integration Capabilities RDS MySQL integrates well with other AWS services like Lambda, CloudWatch, and IAM. It also supports integration with third-party applications, providing flexibility in building diverse applications. Aurora MySQL not only integrates seamlessly with other AWS services but also supports native integration with Lambda, enabling serverless computing. It also supports cross-region replication with RDS for MySQL, increasing its extensibility. While both RDS MySQL and Aurora MySQL provide efficient integration capabilities, Aurora MySQL's native integration with Lambda and support for cross-region replication with RDS MySQL gives it a slight edge when it comes to integration efficiency. Conclusion To summarize, while both AWS RDS MySQL and Aurora MySQL offer robust performance, security, and ease of use, there are key differences. Aurora MySQL stands out with its superior performance, faster data replication, more flexible management, and enhanced integration capabilities. However, RDS MySQL might still be the optimal choice for small to medium-sized workloads, given its cost-efficiency and robust feature set. The decision between AWS RDS MySQL and Aurora MySQL should be made based on your specific needs. If your priority is superior performance, high scalability, and advanced integration capabilities, Aurora MySQL might be the best fit. However, if you're looking for a cost-effective solution for moderate workloads, RDS MySQL might be your go-to option. Ultimately, the choice between RDS MySQL and Aurora MySQL depends on your unique situation. It's important to assess your requirements, workload size, budget, and future growth plans before making a decision. Remember, what works best for one organization may not necessarily work best for another. It's all about aligning your choice with your specific needs and goals.

By Satrajit Basu

CORE

Serverless Image Generation Application Using Generative AI on AWS

Whether it's crafting personalized content or tailoring images to user preferences, the ability to generate visual assets based on a description is quite powerful. But text-to-image conversion typically involves deploying an end-to-end machine learning solution, which is quite resource-intensive. What if this capability was an API call away, thereby making the process simpler and more accessible for developers? This tutorial will walk you through how to use AWS CDK to deploy a Serverless image generation application implemented using AWS Lambda and Amazon Bedrock, which is a fully managed service that makes base models from Amazon and third-party model providers (such as Anthropic, Cohere, and more) accessible through an API. Developers can leverage leading foundation models through a single API while maintaining the flexibility to adopt new models in the future. The solution is deployed as a static website hosted on Amazon S3 accessible via an Amazon CloudFront domain. Users can enter the image description which will be passed on to a Lambda function (via Amazon API Gateway) which in turn will invoke the Stable Diffusion model on Amazon Bedrock to generate the image. The entire solution is built using Go - this includes the Lambda function (using the aws-lambda-go library) as well as the complete solution deployment using AWS CDK. The code is available on GitHub. Prerequisites Before starting this tutorial, you will need the following: An AWS Account (if you don't yet have one, you can create one and set up your environment here) Go (v1.19 or higher) AWS CDK AWS CLI Git Docker Clone this GitHub repository and change it to the right directory: git clone https://github.com/build-on-aws/amazon-bedrock-lambda-image-generation-golang cd amazon-bedrock-lambda-image-generation-golang Deploy the Solution Using AWS CDK To start the deployment, simply invoke cdk deploy. cd cdk export DOCKER_DEFAULT_PLATFORM=linux/amd64 cdk deploy You will see a list of resources that will be created and will need to provide your confirmation to proceed (output shortened for brevity). Bundling asset BedrockLambdaImgeGenWebsiteStack/bedrock-imagegen-s3/Code/Stage... ✨ Synthesis time: 7.84s //.... omitted This deployment will make potentially sensitive changes according to your current security approval level (--require-approval broadening). Please confirm you intend to make the following modifications: //.... omitted Do you wish to deploy these changes (y/n)? y This will start creating the AWS resources required for the application. If you want to see the AWS CloudFormation template which will be used behind the scenes, run cdk synth and check the cdk.out folder. You can keep track of the progress in the terminal or navigate to the AWS console: CloudFormation > Stacks > BedrockLambdaImgeGenWebsiteStack. Once all the resources are created, you can try out the application. You should have: The image generation Lambda function and API Gateway An S3 bucket to host the website's HTML page CloudFront distribution And a few other components (like IAM roles, permissions, S3 Bucket policy, etc.) The deployment can take a bit of time since creating the CloudFront distribution is a time-consuming process. Once complete, you should get a confirmation along with the values for the S3 bucket name, API Gateway URL, and the CloudFront domain name. Update the HTML Page and Copy It to the S3 Bucket Open the index.html file in the GitHub repo, and locate the following text: ENTER_API_GATEWAY_URL. Replace this with the API Gateway URL that you received as the CDK deployment output above. To copy the file to S3, I used the AWS CLI: aws s3 cp index.html s3://<name of the S3 bucket from CDK output> Verify that the file was uploaded: aws s3 ls s3://<name of the S3 bucket from CDK output> Now you are ready to access the website! Verify the Solution Enter the CloudFront domain name in your web browser to navigate to the website. You should see the website with a pre-populated description that can be used as a prompt. Click Generate Image to start the process. After a few seconds, you should see the generated image. Modify the Model Parameters The Stability Diffusion model allows us to refine the generation parameters as per our requirements. The Stability.ai Diffusion models support the following controls: Prompt strength (cfg_scale) controls the image's fidelity to the prompt, with lower values increasing randomness. Generation step (steps) determines the accuracy of the result, with more steps producing more precise images. Seed (seed) sets the initial noise level, allowing for reproducible results when using the same seed and settings. Click Show Configuration to edit these. Max values for cfg_steps and steps are 30 and 150, respectively. Don’t Forget To Clean Up Once you're done, to delete all the services, simply use: cdk destroy #output prompt (choose 'y' to continue) Are you sure you want to delete: BedrockLambdaImgeGenWebsiteStack (y/n)? You were able to set up and try the complete solution. Before we wrap up, let's quickly walk through some of the important parts of the code to get a better understanding of what's going the behind the scenes. Code Walkthrough Since we will only focus on the important bits, a lot of the code (print statements, error handling, etc.) has been omitted for brevity. CDK You can refer to the CDK code here. We start by creating the API Gateway and the S3 bucket. apigw := awscdkapigatewayv2alpha.NewHttpApi(stack, jsii.String("image-gen-http-api"), nil) bucket := awss3.NewBucket(stack, jsii.String("website-s3-bucket"), &awss3.BucketProps{ BlockPublicAccess: awss3.BlockPublicAccess_BLOCK_ALL(), RemovalPolicy: awscdk.RemovalPolicy_DESTROY, AutoDeleteObjects: jsii.Bool(true), }) Then we create the CloudFront Origin Access Identity and grant S3 bucket read permissions to the CloudFront Origin Access Identity principal. Then we create the CloudFront Distribution: Specify the S3 bucket as the origin. Specify the Origin Access Identity that we created before. oai := awscloudfront.NewOriginAccessIdentity(stack, jsii.String("OAI"), nil) bucket.GrantRead(oai.GrantPrincipal(), "*") distribution := awscloudfront.NewDistribution(stack, jsii.String("MyDistribution"), &awscloudfront.DistributionProps{ DefaultBehavior: &awscloudfront.BehaviorOptions{ Origin: awscloudfrontorigins.NewS3Origin(bucket, &awscloudfrontorigins.S3OriginProps{ OriginAccessIdentity: oai, }), }, DefaultRootObject: jsii.String("index.html"), //name of the file in S3 }) Then, we create the image generation Lambda function along with IAM permissions (to the function execution IAM role) to allow it to invoke Bedrock operations. function := awscdklambdagoalpha.NewGoFunction(stack, jsii.String("bedrock-imagegen-s3"), &awscdklambdagoalpha.GoFunctionProps{ Runtime: awslambda.Runtime_GO_1_X(), Entry: jsii.String(functionDir), Timeout: awscdk.Duration_Seconds(jsii.Number(30)), }) function.AddToRolePolicy(awsiam.NewPolicyStatement(&awsiam.PolicyStatementProps{ Actions: jsii.Strings("bedrock:*"), Effect: awsiam.Effect_ALLOW, Resources: jsii.Strings("*"), })) Finally, we configure Lambda function integration with API Gateway, add the HTTP routes, and specify the API Gateway endpoint, S3 bucket name, and CloudFront domain name as CloudFormation outputs. functionIntg := awscdkapigatewayv2integrationsalpha.NewHttpLambdaIntegration(jsii.String("function-integration"), function, nil) apigw.AddRoutes(&awscdkapigatewayv2alpha.AddRoutesOptions{ Path: jsii.String("/"), Methods: &[]awscdkapigatewayv2alpha.HttpMethod{awscdkapigatewayv2alpha.HttpMethod_POST}, Integration: functionIntg}) awscdk.NewCfnOutput(stack, jsii.String("apigw URL"), &awscdk.CfnOutputProps{Value: apigw.Url(), Description: jsii.String("API Gateway endpoint")}) awscdk.NewCfnOutput(stack, jsii.String("cloud front domain name"), &awscdk.CfnOutputProps{Value: distribution.DomainName(), Description: jsii.String("cloud front domain name")}) awscdk.NewCfnOutput(stack, jsii.String("s3 bucket name"), &awscdk.CfnOutputProps{Value: bucket.BucketName(), Description: jsii.String("s3 bucket name")}) Lambda Function You can refer to the Lambda Function code here. In the function handler, we extract the prompt from the HTTP request body and the configuration from the query parameters. Then it's used to call the model using bedrockruntime.InvokeModel function. Note the JSON payload sent to Amazon Bedrock is represented by an instance of the Request struct. The output body returned from the Amazon Bedrock Stability Diffusion model is a JSON payload that is converted into a Response struct that contains the generated image as a base64 string. This is returned as an events.APIGatewayV2HTTPResponse object along with CORS headers. func handler(ctx context.Context, req events.APIGatewayV2HTTPRequest) (events.APIGatewayV2HTTPResponse, error) { prompt := req.Body cfgScaleF, _ := strconv.ParseFloat(req.QueryStringParameters["cfg_scale"], 64) seed, _ := strconv.Atoi(req.QueryStringParameters["seed"]) steps, _ := strconv.Atoi(req.QueryStringParameters["steps"]) payload := Request{ TextPrompts: []TextPrompt{{Text: prompt}, CfgScale: cfgScaleF, Steps: steps, } if seed > 0 { payload.Seed = seed } payloadBytes, err := json.Marshal(payload) output, err := brc.InvokeModel(context.Background(), &bedrockruntime.InvokeModelInput{ Body: payloadBytes, ModelId: aws.String(stableDiffusionXLModelID), ContentType: aws.String("application/json"), }) var resp Response err = json.Unmarshal(output.Body, &resp) image := resp.Artifacts[0].Base64 return events.APIGatewayV2HTTPResponse{ StatusCode: http.StatusOK, Body: image, IsBase64Encoded: false, Headers: map[string]string{ "Access-Control-Allow-Origin": "*", "Access-Control-Allow-Methods": "POST,OPTIONS", }, }, nil } //request/response model type Request struct { TextPrompts []TextPrompt `json:"text_prompts"` CfgScale float64 `json:"cfg_scale"` Steps int `json:"steps"` Seed int `json:"seed"` } type TextPrompt struct { Text string `json:"text"` } type Response struct { Result string `json:"result"` Artifacts []Artifact `json:"artifacts"` } type Artifact struct { Base64 string `json:"base64"` FinishReason string `json:"finishReason"` } Conclusion In this tutorial, you used AWS CDK to deploy a serverless image generation solution that was implemented using Amazon Bedrock and AWS Lambda and was accessed using a static website on S3 via a CloudFront domain. If you are interested in an introductory guide to using the AWS Go SDK and Amazon Bedrock Foundation Models (FMs), check out this blog post. Happy building!

By Abhishek Gupta

CORE

The Easy Way To Deploy Multilanguage Apps to Kubernetes

Kubernetes can be intricate to manage, and companies want to leverage its power while avoiding its complexity. A recent survey found that 84% of companies don’t see value in owning Kubernetes themselves. To address this complexity, Cloud Foundry introduced open-source Korifi, which preserves the classic Cloud Foundry experience of being able to deploy apps written in any language or framework with a single cf push command. But the big difference is that this time, apps are pushed to Kubernetes. In this tutorial, we’ll explore how to use Korifi to deploy web applications written in different languages: Ruby, Node.js, ASP.NET, and PHP. I will also provide insights into Korifi’s functioning and basic configuration knowledge, helping you kick-start your multi-cloud, multitenant, and polyglot journey. Ruby For all the examples in this tutorial, I will use sample web applications that you can download from this GitHub repository, but feel free to use your own. You can also find instructions on installing Korifi in this article, which guides you through the easiest way to achieve that by running two Bash scripts that will set everything up for you. Once you have Korifi installed and have cloned a Ruby sample application, go into the root folder and type the following command: Shell cf push my-ruby-app That’s it! That is all you need to deploy a Ruby application to Kubernetes. Keep in mind that while the first iteration of cf push will take some time as Korifi needs to download a number of elements (I will explain this in the next paragraph); all subsequent runs will be much faster. At any point, if you want to check the status of a Korifi app, you can use the cf app command, which, in the case of our Ruby app, would be: Shell cf app my-ruby-app Node.js Before deploying a Node.js application to Kubernetes using Korifi, let me explain how it works under the hood. One of the key components at play here is Cloud Native Buildpacks. The concept was initially introduced in 2011 and adopted by PaaS providers like Google App Engine, GitLab, Deis, and Dokku. This project became a part of the CNCF in 2018. Buildpacks are primarily designed to convert an application’s source code into an OCI image, such as a Docker image. This process unfolds in two steps: first, it scans the application to identify its dependencies and configures them for seamless operation across diverse clouds. Then, it assembles an image using a Builder, a structured amalgamation of Buildpacks, a foundational build image, a lifecycle, and a reference to a runtime image. Although you have the option to construct your own build images and Buildpacks, you can also leverage those provided by established entities such as Google, Heroku, and Paketo Buildpacks. In this tutorial, I will exclusively use ones provided by Paketo — an open-source project that delivers production-ready Buildpacks for popular programming languages. Let’s briefly demonstrate what Korifi does by manually creating a Buildpack from a Node.js application. You can follow the installation instructions here to install the pack CLI. Then, get into the root folder of your application and run the following command: Shell pack build my-nodejs-app --builder paketobuildpacks/builder:base Your Node.js OCI image is available; you can check this by running the command: Shell docker images Once the Docker image is ready, Korifi utilizes Kubernetes RBAC and CRDs to mimic the robust Cloud Foundry paradigm of orgs and spaces. But the beauty of Korifi is that you don’t have to manage any of that. You only need one command to push a Node.js application to Kubernetes: Shell cf push my-nodejs-app That’s it! ASP.NET Now, let’s push an ASP.NET application. If you run cf push my-aspnet-app, the build will fail, and you will get the following error message: Shell BuildFail: Check build log output FAILED 2023-08-11T19:12:58.11+0000 [STG/] OUT ERROR: No buildpack groups passed detection. 2023-08-11T19:12:58.11+0000 [STG/] OUT ERROR: failed to detect: buildpack(s) failed with err These logs tell us that Korifi may not know a valid Buildpack to package an ASP.NET application. We can verify that by running the following command: Shell cf buildpacks You should get the following output, and we can see that there are no .NET-related buildpacks. Shell position name stack enabled locked filename 1 paketo-buildpacks/java io.buildpacks.stacks.jammy true false paketo-buildpacks/java@9.18.0 2 paketo-buildpacks/go io.buildpacks.stacks.jammy true false paketo-buildpacks/go@4.4.5 3 paketo-buildpacks/nodejs io.buildpacks.stacks.jammy true false paketo-buildpacks/nodejs@1.8.0 4 paketo-buildpacks/ruby io.buildpacks.stacks.jammy true false paketo-buildpacks/ruby@0.39.0 5 paketo-buildpacks/procfile io.buildpacks.stacks.jammy true false paketo-buildpacks/procfile@5.6.4 To fix that, first, we need to tell Korifi which Buildpack to use for an ASP.NET application by editing the ClusterStore: Shell kubectl edit clusterstore cf-default-buildpacks -n tutorial-space Make sure to replace tutorial-space with the value you used during your Korifi cluster configuration. Add the line – image: gcr.io/paketo-buildpacks/python; your file should look like this: Shell spec: sources: - image: gcr.io/paketo-buildpacks/java - image: gcr.io/paketo-buildpacks/nodejs - image: gcr.io/paketo-buildpacks/ruby - image: gcr.io/paketo-buildpacks/procfile - image: gcr.io/paketo-buildpacks/go - image: gcr.io/paketo-buildpacks/python Then we need to tell Korifi in which order to use Buildbacks by editing our ClusterBuilder: Shell kubectl edit clusterbuilder cf-kpack-cluster-builder -n tutorial-space Add the line – id: paketo-buildpacks/dotnet-core at the top of the spec order list. your file should look like this: Shell spec: sources: - image: gcr.io/paketo-buildpacks/java - image: gcr.io/paketo-buildpacks/nodejs - image: gcr.io/paketo-buildpacks/ruby - image: gcr.io/paketo-buildpacks/procfile - image: gcr.io/paketo-buildpacks/go - image: gcr.io/paketo-buildpacks/python If everything was done right, you should see the .NET Core Paketo Buildpack in the list output by the cf buildpacks command. Finally, you can simply run cf push my-aspnet-app to push your ASP.NET application to Kubernetes. PHP We need to follow the same process for PHP with the Buildpack paketo-buildpacks/php that needs to be added to the ClusterStore and ClusterBuilder. For anyone using Korifi version 0.9.0 released a few days ago, the issue that I am about to discuss has been fixed. But in case you are using an older version, running cf push my-php-app will fail and return the following error message: Shell [APP/] OUT php: error while loading shared libraries: libxml2.so.2: cannot open shared object file: No such file or directory The OCI image is missing the libxml library, which is required by PHP, this is probably due to the builder not supporting PHP. To check that, let’s look what builder Korifi is using by running this command: Shell kubectl describe clusterbuilder cf-kpack-cluster-builder | grep 'Run Image' Which will output the following: Shell Run Image: index.docker.io/paketobuildpacks/run-jammy-base@sha256:4cf369b562808105d3297296efea68449a2ae17d8bb15508f573cc78aa3b3772a As you can see, Korifi currently uses Paketo Jammy Base, which, according to its Github repo description, does not support PHP. You also can check that by looking at the builder’s builder.toml file or by running the command pack builder suggest, which will return the output: Shell Suggested builders: [...] Paketo Buildpacks: paketobuildpacks/builder-jammy-base Ubuntu 22.04 Jammy Jellyfish base image with buildpacks for Java, Go, .NET Core, Node.js, Python, Apache HTTPD, NGINX and Procfile Paketo Buildpacks: paketobuildpacks/builder-jammy-buildpackless-static Static base image (Ubuntu Jammy Jellyfish build image, distroless-like run image) with no buildpacks included. To use, specify buildpack at build time. Paketo Buildpacks: paketobuildpacks/builder-jammy-full Ubuntu 22.04 Jammy Jellyfish full image with buildpacks for Apache HTTPD, Go, Java, Java Native Image, .NET, NGINX, Node.js, PHP, Procfile, Python, and Ruby [...] While Jammy Base does not support PHP, the Jammy Full builder does. There are multiple ways to get Korifi to use another builder, I will just cover one way in this tutorial. This way assumes that we used the easy way to install Korifi with the deploy-on-kind.sh script. You need to go to Korifi source code and edit the file scripts/assets/values.yaml so that the fields clusterStackBuildImage and clusterStackRunImage are set to paketobuildpacks/build-jammy-full by running this command: Shell sed -i 's/base/full/g' scripts/assets/values.yaml` Then, run the scripts/deploy-on-kind.sh script. That’s it! Korifi will use the Jammy full builder, and Korifi will be able to deploy your PHP application with a cf push my-php-app command. Summary Hopefully, now you’ve experienced just how easy it is to use Korifi to deploy applications to Kubernetes written in Ruby, Node.js, ASP.NET, and PHP. You can stay tuned with the Korifi project by following Cloud Foundry X account and joining the Slack workspace.

By Sylvain Kalache

Unleashing Linux Power on Windows: Exploring the WSL Feature

The Windows Subsystem for Linux (WSL), which unites the Windows and Linux operating systems, has completely changed how users and developers interact with these systems. WSL, which Microsoft first released in 2016, offers a compatibility layer for Windows that enables users to run native Linux command-line tools and applications on their Windows systems. A whole new world of opportunities has been made possible for both developers and enthusiasts by this potent feature, which has eliminated the gap between two historically separate operating systems. Microsoft revolutionized the relationship between Windows and Linux when it introduced the Windows Subsystem for Linux (WSL) feature. WSL brings the strength, adaptability, and extensive ecosystem of Linux to the Windows operating system by enabling developers and users to run a full-fledged Linux environment directly on Windows. Let us explore the world of WSL, including its advantages, applications, and how it has merged two different platforms. What Is WSL? Windows Subsystem for Linux is known by the initials WSL. Users can directly use native Linux command-line tools and applications on the Windows operating system thanks to a compatibility layer created by Microsoft. Without the use of dual booting or virtual machines, WSL enables programmers, system administrators, and users to collaborate with Linux-based software in a Windows environment without any difficulty. There Are Two Major Versions of WSL WSL 1: In the first version of WSL, the Linux binaries interact with a translation layer provided by the WSL core. This translation layer intercepts Linux system calls and translates them into corresponding Windows system calls. WSL 1 does not include a Linux kernel but relies on the Windows kernel for execution. It provides a Linux-compatible environment and allows users to run Linux command-line tools and applications, access the file system, and execute shell scripts. WSL 2: WSL 2 introduces a significant architectural change by incorporating a lightweight virtual machine (VM). In this version, a full Linux kernel runs within the VM, providing improved compatibility, performance, and support for more Linux-specific features. WSL 2 utilizes the Virtual Machine Platform built into Windows and leverages the Hyper-V hypervisor to run the Linux kernel. It also introduces a more efficient file system protocol (9P) for faster file system access. WSL offers integration with the Windows environment, allowing users to access and work with files seamlessly between Windows and Linux. It supports various Linux distributions, such as Ubuntu, Debian, Fedora, and more, which can be installed directly from the Microsoft Store or by importing custom distributions. WSL also enables users to install and use Linux package managers, run Linux servers, develop and test cross-platform applications, and perform system administration tasks within the Windows ecosystem. Overall, WSL provides a powerful tool for developers and users who need to work with both Windows and Linux environments, offering a convenient and streamlined experience for running Linux software on Windows machines. The WSL Architecture At its core, WSL comprises two distinct versions: WSL 1 and WSL 2. WSL 1 leverages a translation layer that interprets Linux system calls into Windows equivalents, enabling Linux binaries to run on Windows. On the other hand, WSL 2 utilizes a lightweight virtual machine (VM) to run a full Linux kernel, offering improved performance and full system call compatibility. The Windows Subsystem for Linux (WSL) architecture consists of several components that enable the integration of Linux functionalities within the Windows operating system. Here is an overview of the WSL architecture: WSL Core: At the heart of WSL is the WSL core, which is responsible for managing the Linux system call interface and translating Linux system calls into their Windows equivalents. This component provides the necessary compatibility layer that allows Linux binaries to run on Windows. WSL Distro: A WSL distribution, such as Ubuntu, Debian, or Fedora, is a package that includes a root file system containing the Linux user space environment, libraries, and binaries. Each WSL distribution runs within its own lightweight virtual machine (VM) or a compatibility layer depending on the version of WSL being used. WSL 1: In WSL 1, the WSL core operates as a translation layer that intercepts Linux system calls made by Linux binaries and translates them into Windows system calls. It does not include a Linux kernel and relies on the Windows kernel for execution. The file system is accessed through the DrvFs file system driver, which provides translation between Linux and Windows file systems. WSL 2: WSL 2 introduces a significant architectural change by utilizing a lightweight VM to run a full Linux kernel. In this version, the WSL core interacts with the Linux kernel directly, resulting in improved compatibility and performance compared to WSL 1. The file system is accessed through the 9P protocol, allowing for faster file system operations. Virtual Machine Platform: WSL 2 utilizes the Virtual Machine Platform, which is a lightweight virtualization technology built into Windows. This platform hosts the virtual machine that runs the Linux kernel within WSL 2. It provides isolation between the Linux kernel and the Windows host, enabling better compatibility and performance. Windows Kernel: The Windows kernel forms the underlying foundation of the WSL architecture. It provides the necessary system services and resources required for WSL to function, including hardware access, process management, and file system operations. Windows Console: The Windows Console is the terminal interface for interacting with WSL. It provides the command-line interface (CLI) where users can execute Linux commands, run Linux applications, and manage their WSL environment. The Windows Console supports various terminal emulators and can be customized with different shells and tools. The WSL architecture allows for seamless integration between the Windows and Linux environments, enabling users to leverage Linux tools, utilities, and applications within the Windows operating system. Whether running Linux binaries through the WSL core in WSL 1 or utilizing a lightweight virtual machine with a full Linux kernel in WSL 2, WSL provides a bridge between two traditionally distinct operating systems, expanding the possibilities for developers, system administrators, and users. WSL Versions and Features WSL 1 was the initial release, featuring a compatibility layer translating Linux system calls to the Windows kernel. While it provided substantial improvements over traditional compatibility layers, it lacked the full Linux kernel functionality. WSL 2, introduced with the Windows 10 May 2020 Update, brought significant enhancements by employing a lightweight virtualization technology. This update replaced the compatibility layer with a complete Linux kernel, running in a lightweight virtual machine, thus delivering a more authentic Linux environment and improved performance. Benefits The Windows Subsystem for Linux (WSL) offers numerous benefits to users, developers, and system administrators. Here are some of the key advantages of using WSL: Seamless Integration: WSL seamlessly integrates Linux and Windows environments, allowing users to run Linux command-line tools and applications directly on their Windows machines. This eliminates the need for dual booting or running virtual machines, streamlining the development and execution of cross-platform projects. Compatibility and Portability: WSL ensures compatibility between Windows and Linux by providing a Linux-compatible environment within Windows. This enables developers to write, test, and run Linux-specific code or scripts on their Windows machines, ensuring the applications work smoothly across different platforms. It also facilitates easier collaboration among teams with diverse operating system preferences. Access to Linux Ecosystem: WSL provides access to the vast Linux ecosystem, allowing users to install and run various Linux distributions, such as Ubuntu, Debian, and Fedora, directly from the Microsoft Store. This enables users to leverage the extensive range of software, development frameworks, and libraries available in the Linux community, enhancing their development capabilities. Improved Development Environment: WSL enhances the development environment on Windows machines. Developers can utilize familiar Linux tools, utilities, and workflows, such as bash, grep, sed, awk, and package managers like apt and yum, without leaving the Windows ecosystem. This flexibility improves productivity and enables developers to leverage the strengths of both operating systems. Docker Integration: WSL greatly enhances Docker workflows on Windows. With WSL 2, users can run Docker containers natively, resulting in improved performance and eliminating the need for resource-intensive virtualization solutions like Hyper-V. This allows developers to seamlessly work with Dockerized applications, enabling efficient container-based development and deployment. System Administration and Troubleshooting: WSL offers a powerful environment for system administrators and IT professionals. It allows them to leverage Linux-oriented tools, scripts, and utilities to manage and troubleshoot Windows systems effectively. They can perform tasks such as scripting, network diagnostics, and automation using the vast array of Linux tools available within WSL. Learning and Skill Development: WSL serves as an excellent learning tool for individuals seeking to gain familiarity with Linux or enhance their Linux skills. It provides a risk-free environment for experimenting, practicing, and acquiring proficiency in Linux command-line operations, scripting, and administration, all within the comfort of a Windows machine. Performance and Resource Efficiency: With the introduction of WSL 2, which utilizes a lightweight virtual machine, users can experience improved performance and resource efficiency compared to WSL 1. The full Linux kernel running within the virtual machine ensures better system call compatibility and enhanced execution speed for Linux applications. Enhanced File System Integration: WSL seamlessly integrates Windows and Linux file systems, enabling easy access to files and directories across both environments. This allows users to work on files using their preferred Windows or Linux tools without the need for complex file sharing or conversion processes. Use Cases Web Development: WSL is widely used by web developers who work with both Windows and Linux stacks. It allows them to seamlessly switch between Windows-based development tools like Visual Studio and Linux-based servers, ensuring consistency and minimizing the need for separate development environments. Developers can run popular web development tools, such as Node.js, Nginx, and Apache, directly within WSL, enabling them to build and test web applications efficiently. System Administration: WSL provides system administrators with a powerful toolset to manage Windows-based systems while leveraging their Linux expertise. Administrators can utilize Linux-specific command-line tools and scripts to perform various system administration tasks, such as network configuration, package management, and troubleshooting. This allows for efficient system management and automation within the Windows environment. Data Science and Machine Learning: WSL has gained popularity among data scientists and researchers in the field of machine learning. It allows them to run Linux-based frameworks, such as TensorFlow, PyTorch, and scikit-learn, seamlessly on their Windows machines. Data scientists can leverage the computational power of their Windows hardware while accessing the rich ecosystem of data science libraries and tools available in Linux. WSL enables them to develop and experiment with machine learning models, process large datasets, and perform data analysis tasks efficiently. DevOps and Continuous Integration/Continuous Deployment (CI/CD): WSL is a valuable asset for DevOps teams working with mixed operating system environments. It enables seamless integration and collaboration between developers using different platforms. With WSL, developers can test and deploy applications built for Linux environments directly on their Windows machines, ensuring compatibility and reducing deployment issues. WSL can be combined with popular CI/CD tools like Jenkins, GitLab CI/CD, or Azure DevOps, enabling efficient build and deployment pipelines for cross-platform applications. Education and Learning: WSL serves as an excellent learning tool for individuals interested in exploring Linux and acquiring Linux skills. It provides a safe and accessible environment for students, enthusiasts, and newcomers to practice and experiment with Linux command-line operations, scripting, and system administration. WSL’s integration within Windows simplifies the learning process by eliminating the need for separate hardware or virtual machines. Cross-Platform Development: WSL enables developers to create cross-platform applications with ease. By utilizing the Linux environment within WSL, developers can ensure their applications work seamlessly on both Windows and Linux operating systems. They can test and debug their code in the Linux environment, ensuring compatibility before deployment. Software Testing: WSL offers a convenient platform for software testing. Testers can utilize WSL to run automated tests, perform compatibility testing, and validate software behavior in a Linux environment while working on a Windows machine. This allows for efficient testing and debugging without the need for dedicated Linux hardware or virtual machines. Research and Experimentation: WSL provides researchers and enthusiasts with a flexible platform for experimentation and prototyping. Whether it’s exploring new technologies, testing novel software configurations, or conducting academic research, WSL’s compatibility and access to the Linux ecosystem enable researchers to work in a familiar and versatile environment. Limitations and Future Developments While WSL has brought remarkable Linux integration to Windows, it’s essential to acknowledge its limitations. Not all Linux applications or graphical user interfaces (GUIs) are fully compatible with WSL. Additionally, certain hardware or system-level functionality may not be accessible within the WSL environment. Microsoft is actively working on improving WSL with regular updates and feature enhancements. Developers can expect further improvements in hardware support, increased file system performance, and better integration with Windows-specific features. While WSL offers significant advantages, it does have limitations, including GUI support and certain performance considerations. However, Microsoft is actively addressing these limitations and investing in future developments for WSL. With planned improvements in GUI application support, file system performance, kernel updates, integration with Windows development tools, and enhanced networking and GPU support, WSL is poised to become an even more powerful and versatile tool for bridging the gap between Windows and Linux environments. While the Windows Subsystem for Linux (WSL) provides significant benefits, it does have some limitations. Additionally, Microsoft continues to invest in WSL and has plans for future developments. Let’s explore the limitations and potential future advancements for WSL: Limitations of WSL Graphical User Interface (GUI) Limitations: WSL primarily focuses on providing a command-line interface, and running Linux GUI applications directly within WSL is not supported. While there are workarounds available, such as running an X server on Windows, the GUI experience can be limited and may not offer the same level of integration as native applications. Performance Considerations: While WSL 2 offers improved performance compared to WSL 1, it still introduces a layer of translation between Linux system calls and Windows equivalents. In scenarios where performance is critical, such as heavy computational workloads or high I/O operations, running native Linux or virtualized Linux environments may provide better performance. Kernel Compatibility: WSL provides a Linux-compatible environment but does not offer a complete Linux kernel. This means that certain kernel-specific features or behaviors may not be available or behave differently within WSL. While WSL aims to provide a broad compatibility range, some Linux applications or system components may have specific kernel requirements that are not met within WSL. Limited Linux Kernel Access: WSL operates as a lightweight virtual machine (VM) with its own Linux kernel, isolated from the host Windows kernel. This isolation can limit direct access to hardware resources or kernel-level functionalities that require direct interaction with the host kernel. It may impact scenarios where deep integration with hardware or specialized kernel features is necessary. Future Developments for WSL GUI Application Support: Microsoft has acknowledged the demand for running Linux GUI applications within WSL and is actively working on improving the graphical capabilities of WSL. In the future, we can expect better support for running Linux GUI applications directly within the Windows environment, providing a more integrated and seamless experience. File System Performance Enhancements: Microsoft has identified file system performance as an area for improvement within WSL. Future developments aim to optimize file system operations, reduce latency, and improve overall performance when accessing files between the Windows and Linux environments. Kernel Updates: Microsoft is committed to keeping WSL up to date with the latest Linux kernel improvements. This includes supporting newer kernel versions, enhancing system call compatibility, and incorporating bug fixes and security patches to ensure a more robust and reliable Linux environment within WSL. Integration with Windows Development Tools: Microsoft is working on tighter integration between WSL and Windows development tools, such as Visual Studio and Windows Terminal. This integration aims to provide a more seamless experience for developers, allowing them to seamlessly switch between Windows and Linux development workflows within a unified environment. Enhanced Networking and GPU Support: Microsoft is exploring ways to improve networking capabilities within WSL, enabling more advanced networking scenarios and smoother integration with Windows networking features. Additionally, there are plans to improve GPU support within WSL, allowing Linux applications to leverage the full power of dedicated GPUs on Windows machines. Conclusion The Windows Subsystem for Linux (WSL) has revolutionized the development landscape by enabling developers to seamlessly merge the power of Linux with the familiarity of Windows. Without giving up the Windows operating system, it offers users a rare chance to learn about Linux, access its robust ecosystem, and utilize its tools and utilities. WSL promises an even more feature-rich and robust experience as it continues to advance and update, bridging the gap between Windows and Linux like never before. Windows users now interact with Linux tools and applications in a completely new way thanks to the Windows Subsystem for Linux (WSL). WSL has made it possible for programmers, system administrators, and enthusiasts to collaborate easily in a hybrid environment that combines the benefits of both Windows and Linux by bridging the gap between the two operating systems. WSL has developed into an essential tool for everyone thanks to its compatibility, flexibility, and access to the Linux ecosystem. Developers, system administrators, researchers, and students can all work effectively in a hybrid Windows-Linux environment thanks to the Windows Subsystem for Linux (WSL), which has a wide range of use cases. WSL provides an adaptable platform that combines the best of the Windows and Linux ecosystems for a variety of uses, including web development, system administration, data science, DevOps, and education. For people and teams working in a variety of operating system environments, WSL is a useful tool thanks to its seamless integration, compatibility, and access to Linux tools and libraries.

By Aditya Bhuyan

Tools

DZone's Featured Tools Resources

Top Tools Experts

The Latest Tools Topics