A common data development task is to move data from a target to a source. It can be a time consuming and tedious process when you have many sources or many targets of different types. This task can be made more complex when you add an automation component. These tasks can be made much quicker and easier to manage when using the right tool, for example Linx.
This article will look at exactly how you can read and write data from these different sources. Each source has a video that describes exactly how data loading is done. Topics of discussion include:
- Read data from a CSV or Text File
- Read data from Excel files
- Load data from an API
- Read data from a JSON or XML File
- Database-to-database interactions
- Automating your file-loading process with ease
- Writing data to a CSV or Text file
- Writing data to an Excel file
- Write data to JSON
- Distributing your data via REST APIs
- Easily distribute created files via Email
You can move data from any source into any target in the following ways:
Load Data
Loading data from a CSV/Text File
You can load data from a CSV or text file quickly and easily by using the File plugin. With the ReadText component, you can set the file to be read either statically or dynamically, automatically create the field mapping by reading them from the file and then extract the data. This process can be quickly set up as it is managed by a single component, and once you have the data you can apply validations, transformations and move it to any target. The below video shows how this can be done in a a few minutes:
Load data from an Excel file
Loading data from an Excel file can be complicated and tricky with code. There are many ways to solve this problem, but the quickest has got to be to use a tool to load the data. With Linx you can extract data from an Excel file by specifying what file is to be read (static or dynamically), set the range to be read, and that’s it, you can extract the data. You can then access each row or handle the entire dataset. Once data is extracted, you can do anything with it, such as apply transformations and validations, merge it with other data and then load it into any target. The video below illustrates how this can be done:
Load data from any API
Loading data from an API is a common task, as nearly every modern system has a REST API ready and available. It offers an easy and clean integration mechanism. This can be quickly done by using the CallRESTEndpoint component in Linx that allows you to call any endpoint. Once you have called the endpoint, you can manipulate the data however you choose by using Types. Types allow you to create custom data objects to handle data objects and access specific fields in the data structure. The data can then be used in any way that you need it to be used, for example, loaded into a SQL database. The video below shows how you can quickly read from a REST API:
Read from XML/JSON Files
Reading data from an XML file can come with its challenges. These challenges can be overcome by using the right tool. Linx allows you to read a file, extract the XML data and then parse that XML to access each individual field so that you can use and manipulate that data according to your needs.
The video below shows how an XML file is read and that data is then loaded into a database:
This video shows how data can be easily loaded from a JSON File and how JSON objects are managed in Linx. Data is loaded from a JSON file into a database
Database-to-database interactions
If you need to move data from one database to another, you have many options to consider. You may need to move data from a MySQL database to a SQL Server database or vice versa. This can be easily done by using a platform tool like Linx. In Linx you have a wide variety of database components that can allow you to facilitate your Data Migration or Data Integration needs. You can connect to any database (MSSQL, Oracle, PostgreSQL or MySQL). The below video will show you how to move data from one database to another:
Automated file loading
You may want to read a file when it is dropped into a folder. This type of automated loading can be complicated to configure in code, but in Linx you can quickly set this up by using the DirectoryWatch service. This service will monitor a directory for any changes, such as when a file is created. You can then load that file by using the TextFileRead Component. This can also be used to load Excel, binary or any other file type. The video below shows how this can be quickly configured:
Write Data
Writing data to a CSV/Text file
You may need to write data extracted from another source into a CSV or text file. While this is generally a quick and easy task to do, it can become tedious when you have many files to create or when you need to apply advanced transformations and logic to data before creating the CSV file. A low-code tool like Linx can greatly assist in speeding up this process and facilitate the creation of CSV/text files in an automated fashion. The video below shows how a CSV file can be created from data extracted from a database:
Writing data to an Excel file
Often you may want to create and write data to an Excel file. This can be a complicated process but when using a low-code platform like Linx it can be done by configuring a single component. You can also load data from any source, combine data, and apply transformations before creating the Excel file. The below video shows how you can create an Excel file with data from a database:
Create a JSON file
You may need to create a JSON file. This can be quickly done by using the TextFileWrite component. Because Linx natively handles data objects as JSON, you can quickly create a JSON file. The below video shows how this can be done:
Create a REST API to host your data
You may need to distribute your data via API. In this case, it is important to use a tool that will allow you to create an API that is both flexible and easy to host. For example, with Linx, you can create fully customizable, flexible, bespoke APIs to host your data. You can create the API by providing your own Open API 3.0 specification or by using a Wizard to define the endpoints in two ways. You are also in control of security and all underlying logic. The below video shows how APIs are created in Linx:
Distributing data via Email
You may need to distribute reports, generated files or simply data extracts via Email. This can be done in Linx by using the Email plugin. Emails are sent via SMTP mail server, and you can either send your data in the email body or you can attach generated files as attachments to the email. The below video shows how this can be done in Linx: